Sunday, 21 October, 2018 UTC

Using N1QL with Couchbase Eventing Functions.

Using N1QL with Couchbase Eventing Functions.

Summary

Now, this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning. — Winston Churchill
Updating data is usually not the end, but usually a progress of a workflow.  Shipping follows ordering; inventory update follows shipping; adjusting credit follows returning;  The next step in the process is required to act to keep the workflow going. The workflows can be simple with few steps or complex with hundreds of steps.  Business Process Management (BPM) is an industry by itself.
Couchbase 5.5 introduced Evening service. Developers can write a Javascript function to execute upon a change to the data. We refer inserts, Updates, Merges, and Deletions together as mutations. Multiple specific use cases have been documented for developing these eventing functions.
JSON data model in Couchbase came from JavaScript.  N1QL is SQL for JSON. The eventing functions are written in Javascript and has integrated N1QL. Using the Eventing functions, writing procedural business logic with instant access to data easy.
// Simple counting function
// After every update to travel sample, keep track of the total number of documents of each type.
function OnUpdate(doc, meta) {
    try {
   
    // Issue SELECT statement to get the counts.
    var ginfo = SELECT type, COUNT(type) typecount FROM `travel-sample` WHERE `type` IS NOT MISSING GROUP BY type;

    // loop through the resultset
    for (var val of ginfo) {
      var ckey = "trcount" + val.type;  // Create the document key-string.
      var vtype = val.type; var vtc = val.typecount;  // get the type, count
      // do the update.
      UPDATE T1 USE KEYS[$ckey] SET  type =  $vtype, typecount =$vtc;
    }
  } catch(e) { log(e); }    
}
Here’s the lifecycle of the functions from a developer perspective. For every mutation, you can define any number of these functions to be executed. It’s the developer responsibility to size the execution times of the functions, depending on the number of mutations.
Here’s the lifecycle of the Eventing functions:
For every insert, update or delete (direct or expiry), you can execute one or more javascript functions.  These functions can read the new data and type of action and then execute the subsequent action. This functionality is well described in Couchbase blogs, and articles.
Statement Type Eventing Function Invoked
SELECT None
INSERT OnUpdate().  The function is invoked once per document inserted. Simple insert inserts a single document. Inserts can have multiple documents using multiple documents in the VALUES clause or can insert multiple documents via INSERT INTO…SELECT statement.
UPDATE OnUpdate() is invoked once per document updated, except when multiple updates on the same document is deduped into a single update. Update statement can update multiple documents.
UPSERT OnUpdate().  The behavior is similar to INSERT.
DELETE OnDelete(). Invoked once per documented deleted.
MERGE OnUpdate() and/or OnDelete() depending on the insert, update, and delete actions.
CREATE INDEX, DROP INDEX, EXPLAIN, PREPARE, GRANT, REVOKE No eventing function is invoked.
EXECUTE Depends on the type of the statement executed.
These functions can also execute N1QL statements.  Rest of the article looks at all aspects of N1QL executed in Eventing Functions.
N1QL Statements in Eventing functions.
Statement Use cases for N1QL statements in Eventing Functions
CREATE INDEX Since the schema is flexible, you could potentially inspect the data often/periodically to detect new fields and then create indexes on it.
DELETE Cascade deletes.
DROP INDEX A corollary to the CREATE INDEX use case.
INFER Periodic introspection of the bucket for the structure.  Then take action (validate, create index, update the data model) if necessary.
INSERT Maintaining referential data (eventually).
Updating other documents with references to this data.  E.g Data from a new zip, state, etc.
Data copy (fully or partially) to secondary/tertiary documents. Similar to post-trigger action.
MERGE Keep the secondary data in sync.
SELECT Fetch any data, run any reports periodically, etc.
Check for various things like data quality, data validity,
When you do know the target document key, use the built-in direct document references. See examples at: https://docs.couchbase.com/server/5.5/eventing/eventing-examples.html
UPSERT Keeping secondary/tertiary data in sync.
Similar to post-trigger action.
UPDATE Keeping secondary/tertiary data in sync.
Similar to post-trigger action.
Examples:  Let’s try some eventing functions with N1QL.
select meta().id, * from execute S1;

[
  {
    "S1": {
      "name": "Joe Smith",
      "zip": "94501"
    },
    "id": "Joe::94040"
  },
  {
    "S1": {
      "name": "John Smith",
      "zip": "94040"
    },
    "id": "John::94040"
  }
]



select meta().id, * from S2;


[
  {
    "S2": {
      "name": "Joe Smith",
      "zip": "94501"
    },
    "id": "Joe::94040"
  },
  {
    "S2": {
      "name": "John Smith",
      "zip": "94040"
    },
    "id": "John::94040"
  }
]


select meta().id, * from T1;
<No data>
  1. Simple N1QL in Functions for logging and cascade delete.
Save away every deleted document in a separate bucket.
function OnUpdate(doc, meta) {
}
function OnDelete(meta) {
  try {
      var myid = meta.id;
      //Insert into the bucket of 
      INSERT INTO T1 VALUES(UUID(), {"type" : "deleted", "docid":$myid});
      //Cascade delete
      DELETE FROM S2 USE KEYS [$myid];
  } catch(e) { log(e); }
}

After creating the function above, the application Runs the following:

DELETE FROM S1 WHERE zip = "94040";
// One document was deleted.

SELECT meta().id, * from S1;
// one document just got deleted. There should be only one document remaining.
[
  {
    "S1": {
      "name": "Joe Smith",
      "zip": "94501"
    },
    "id": "Joe::94040"
  }
]



SELECT meta().id, * from T1;
// We should see a log of the deleted
[
  {
    "T1": {
      "docid": "John::94040",
      "type": "deleted"
    },
    "id": "2dc9b33d-3cd4-422e-af9c-b0c664c4660f"
  }
]

SELECT meta().id, * FROM S2;
// We should only see one document due to the effect of cascade delete from the the function
[
  {
    "S2": {
      "name": "Joe Smith",
      "zip": "94501"
    },
    "id": "Joe::94040"
  }
]
  1. OnUpdate() function to keep the aggregate information ready periodically.
function OnUpdate(doc, meta) {
    try {
        var stattime = SELECT lastupdate FROM T1 USE KEYS ["trstattime"];
        for (var t of stattime) {
            var lt = t.lastupdate;
            var d = new Date();
            var n = d.getTime();
            // Every 10 minutes or more
            if ((n - lt)  > (1000 * 60 * 10)) {  
                UPDATE  T1 USE KEYS["trstattime"]  SET lastupdate = NOW_MILLIS() ;
                var ginfo = SELECT type, count(type) typecount FROM `travel-sample` WHERE `type` IS NOT MISSING GROUP BY type;
                for (var val of ginfo) {
                   var ckey = "trcount::" + val.type;
                   var vtype = val.type; var vtc = val.typecount;
                   UPDATE T1 USE KEYS[$ckey] SET  type =  $vtype, typecount = $vtc;
                }
        }
     }
    } catch(e) { log(e); }    
}
References:
  1. Couchbase documentation: https://docs.couchbase.com/server/5.5/eventing/eventing-overview.html
  2. Couchbase blogs on eventing: https://blog.couchbase.com
The post Using N1QL with Couchbase Eventing Functions. appeared first on The Couchbase Blog.

... more @ blog.couchbase.com

blog.couchbase.com