Paul Roub

A Software Tool Geek in His Natural Habitat

Using Zumero for SQL Server With FMDB and SQLCipher

Recently, I described how to use SQLCipher with Zumero to encrypt your locally-synched DBfiles. That was all fine, except:

  1. This was before the advent of Zumero for SQL Server.
  2. The ZSS framework includes a stripped-down, synch-only object model, allowing you to use your iOS SQLite wrapper of choice (I choose FMDB)
  3. There’s a lot of “install this”, “drag this to that”, “install the other thing”… Tedious to describe, tedious to follow.

These days, the smart and sane are using Cocoapods to manage the components they’re using in their Xcode projects. There’s a pod for FMDB, one for SQLCipher and now, a Zumero for SQL Server Cocoapod.

How do we combine all the pieces? Head to your XCode iOS project’s folder, and make sure you have a Podfile. In that Podfile, mention Zumero and FMDB, using the SQLCipher variants of both pods:

platform :ios, '7.0'
pod 'ZumeroSync/SQLCipher'
pod 'FMDB/SQLCipher'

No need to specify SQLCipher itself — it’s already noted as a dependency in both the Zumero and FMDB pods.

Now run pod install (or pod update if other pods are already in place):

$ pod install
Analyzing dependencies
Downloading dependencies
Installing FMDB (2.1)
Installing SQLCipher (2.1.1)
Installing ZumeroSync (1.0.0.1556)
Generating Pods project
Integrating client project

[!] From now on use `EncryptedDBApp.xcworkspace`.

Do as the nice program says, and open the workspace it’s created for you.

Look! Pods!

FMDB, SQLCipher, and Zumero pods installed in project

Include paths are all taken care of:

autocomplete on Zumero inclusion

As for actually using the three?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
// you'll think harder about this
NSString *dbpath = @"/tmp/foo.db";

// creates the local database, if needed
FMDatabase *db = [FMDatabase databaseWithPath:dbpath];

NSString *key = @"shhhh! secret!";
[db setKey:key];

// No networking on the UI thread. Your app thanks you.
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
    NSError *err = nil;

    // synch local/remote changes
    // initialize the local db if needed
    BOOL ok = [ZumeroSync Sync:dbpath
                     cipherKey:key
                     serverUrl:@"https://myserver.example.com"
                        remote:@"remoteFoo"
                    authScheme:nil
                          user:nil  // again, you'll think harder
                      password:nil
                         error:&err];

    if (ok)
    {
        ok = [db open];
        // and off you go
    }
});

Easy to blog, easy to follow.

Using SQLCipher Encryption With the Zumero iOS Framework

A developer recently asked us if, while using Zumero to sync application data to his devices, he could also encrypt that mobile data using SQLCipher.

The short answer: yes, and it’s easy. Zumero and SQLCipher work independently, unaware of each other; there’s no need to worry about SQLCipher during Zumero operations.

You’ll need to take care to set up your projects correctly, and add one SQL statement when opening a database. That’s it. Fully-encrypted local databases, happily synching with remote counterparts.

Setting Up Your Project

Create your XCode iOS project as you normally would. Don’t add any extra libraries at this time. In particular, don’t add sqlite3.

Adding SQLCipher to your project

Using either SQLCipher Community Edition or SQLCipher Commercial Edition (easier to install, faster builds, up-to-date OpenSSL), install SQLCipher and add it to your project per the instructions.

Take a moment and Build your project; it’s easy to skip a step or get a search path wrong. I’ve heard. From a friend. Who’s totally not me.

Adding Zumero to your project

Download the Zumero Client SDK from our Dev Center, and unzip it to a non-temporary location. You’ll want to hang on to the libraries and documentation therein.

Drag ios/Zumero.Framework from the SDK folder into your project’s “Frameworks” folder. No need to copy the files.

copying files to iOS project

Zumero Framework in project

As mentioned in the Zumero iOS documentation, you’ll need to include iOS’s libz and CFNetwork libraries for Zumero to work properly.

When all is said and done, your library list will look something like:

Project library list

…only with less red, since you will have already done your build troubleshooting. Good for you.

Build again, to confirm that Zumero is happily in place.

Encrypting your Zumero databases

Creating a new, encrypted database for Zumero takes one additional step: after opening the database with ZumeroDB::open, you’ll need to issue a SQLite pragma statement to set your encryption key. Use the same key every time you open the database.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
NSError *err = nil;
BOOL ok = YES;

ZumeroDB *db = [[ZumeroDB alloc] initWithName:@"mydb"
                                 folder:nil
                                 host:@"http://localhost:8080"];

if (! [db exists])
  ok = [db createDB:&err];   

if (ok)
  ok = [db open:&err];

if (ok)
{
  NSString *sql = @"pragma key = 'secretword';";
  ok = [cdb execSql:sql values:nil error:&err];
}

You’ll probably want to manage your keys a bit more carefully than that. Life’s so simple, here in sample-app land.

That’s it; your mobile database is encrypted, and can’t be read or modified without that same key. You can, and perhaps should, use different encryption keys for each mobile device. SQLCipher encryption does not apply to the Zumero sync traffic (although of course the communications themselves are encrypted).

Encrypting a plaintext database can’t be done in-place; you should be able to use sqlcipher_export, then replace the old database with the newly-encrypted copy.

Handling Simultaneous Updates in Zumero: There Is No Step 2

When we talk about the advantages of always-local data for mobile apps, the typical first reaction is “that sounds awesome”.

The next step is to start thinking about your apps, and how mobile users interact with your data, and how those users aren’t going to sit down and coordinate with each other:

“It’s my turn to update the inventory records for store #42”

“Done with those yet?”

“Yep, go ahead and sync.”

“Cool, thanks. Now I can update the ordered-quantity data.”

(a) That’s not how work gets done, (b) computers are supposed to take care of picky details for us, and (c) I don’t want to hang out with those guys at all.

Dealing with this scenario in a Zumero-enabled app boils down to:

  • Users update and add data as they see fit.
  • At some point, probably in the background, the app syncs to the server.
  • Up-to-date, merged data now lives on the device.

You don’t have to write any code that knows conflicting changes happened. Seriously. That’s our problem.

Even when the changes are tricky.

Consider the Zumero wiki sample app. A wiki is a collaborative tool. People can, will, should edit the same pages. Sometimes at the same time.

What happens if they do? Andy creates a page on his iPad:

Colin pulls up the app on his iPhone, syncs, and gets his own copy. While offline, he adds a line to the page:

At the same time, Andy completes one of the items on his list, and marks that line as completed.

Colin comes back online, his app checks in and syncs, and he sees this:

Here’s the code that caused their apps to reconcile the conflicting updates:

Sync a Zumero database link
1
ok = [_db sync:scheme user:username password:password error:&err];

There’s no step 2.

To be fair, there were a couple of lines of code that ran when we initially set up the database, which boiled down to:

  1. If there are conflicting edits on the ‘text’ field of a wiki page, try to merge them, Zumero. Thanks.

No step 2.

Zumero: Background Sync in Objective-C

Mobile offline RSS reader, Part 5

In the second “crossover episode” of our series (previous episodes begin on Eric’s blog), we look at background operations.

Previously, we built Zumero databases full of RSS lists and feed contents. The ZumeroReader sample app pulls, reads and displays those feeds on iOS devices. In action, it looks like this:

At startup, it pulls the feed list like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
ZumeroDB *db = [[ZumeroDB alloc]
              initWithName:@"all_feeds"
              folder:path host:host];
db.delegate = self;

NSError *err = nil;
BOOL ok = YES;

if (! [db exists])
  ok = [db createDB:&err];

ok = ok && ([db isOpen] || [db open:&err]);

NSDictionary *scheme = nil;
NSString *uname = nil;
NSString *pwd = nil;

ok = ok && [db sync:scheme user:uname password:pwd error:&err];

We’re creating a Zumero database file named “all_feeds”. It will sync with a similarly-named dbfile on the server side.

We don’t need any authentication info, since (as previously discussed) the feed lists and feed databases are pullable by anyone.

The ZumeroDB::sync method always performs its network activity on a background thread, calling delegate methods when the action is completed (hence db.delegate = self).

In this case, on sync success, we grab the latest copies of the individual feed databases:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
ok = [db selectSql:@"select feeds.feedid, url, title from feeds, about "
    "where (feeds.feedid = about.feedid)" values:nil
    rows:&rows error:&err];
// ...
  
for (NSDictionary *row in rows)
{
  NSNumber *id = [row objectForKey:@"feedid"];
  
  [self syncFeed:id];
}

// ...

- (void) syncFeed:(NSNumber *)feedid
{
  // ...
  NSString *dbname = [NSString stringWithFormat:@"feed_%@", feedid];
  ZumeroDB *db = [[ZumeroDB alloc] initWithName:dbname folder:path host:host];
  // ...
  db.delegate = self;
  ok = [db sync:nil user:nil password:nil error:&err];
  // ...
}

But what if feeds are updated while the app is running? There are seemingly infinite strategies for launching background tasks in iOS, and you’ll use the one that fits your application best. In this case, I cribbed a simple plan from a StackOverflow post - every time a touch is detected, we restart a timer. If that timer actually manages to expire, then we’ve seen no touches in 5 seconds.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
- (void)sendEvent:(UIEvent *)event {
    [super sendEvent:event];
  
    NSSet *allTouches = [event allTouches];
    if ([allTouches count] > 0) {
        UITouchPhase phase = ((UITouch *)[allTouches anyObject]).phase;
        if (phase == UITouchPhaseBegan || phase == UITouchPhaseEnded)
            [self resetIdleTimer:maxIdleTime];
    }
}

- (void)resetIdleTimer:(NSTimeInterval)secs
{
    if (idleTimer) {
        [idleTimer invalidate];
      idleTimer = nil;
    }
  
    idleTimer = [NSTimer scheduledTimerWithTimeInterval:secs
               target:self selector:@selector(idleTimerExceeded)
               userInfo:nil repeats:NO];
}

When that timer fires, we check to see if its been 5 minutes since our last sync, and that a sync is wanted. If so, we sync again:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
- (void)idleTimerExceeded {
  if (wantToSync)
  {
      NSTimeInterval since = [nextSync timeIntervalSinceNow];
      
      if (since <= 0)
      {
          wantToSync = FALSE;
          
          BOOL ok = FALSE;

          // the view controller's sync method from earlier          
          if (mvc)
              ok = [mvc sync];
          
          if (ok)
              self.networkActivityIndicatorVisible = YES;
          else
              // the sync call failed; try again later
              [self waitForSync:(10 * 60)];
      }
      else
      {
          // nope, check again next idle time
          [self resetIdleTimer:since];
      }
      
      return;
  }
  
  [self resetIdleTimer:maxIdleTime];
}

We also kill our timers when exiting, restart them when waking up or activating, etc.

Finally, when we go into the background, we try for one last sync before our process is suspended:

1
2
3
4
5
6
7
8
9
10
11
12
13
- (void)applicationDidEnterBackground:(UIApplication *)application
{
  // ...
  bgtask = [application beginBackgroundTaskWithExpirationHandler:^{
        [self endBackgroundTask:bgtask];
        bgtask = UIBackgroundTaskInvalid;
    }];
    dispatch_async(dispatch_get_global_queue(
      DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
      [mvc sync];
      // finishBackgroundTask will be called by the sync handlers
    });
}

Like any good sample app, there’s so much that this app doesn’t do. You’re invited to experiment with adding them.

  1. We don’t maintain any read/unread information, either at the feed level or for individual items. This would be a good place to try creating/syncing new databases.
  2. We naïvely grab the first <link> element we see and assume that it’s our permalink, without ever checking its rel attribute or content type.
  3. A feed’s contents are displayed as one big chunk of HTML. Individual table cells might be nice.
  4. We don’t make any attempt to become a long-running background process, since we don’t actually play audio, collect location information, etc. Might be fun to play with that.
  5. Configuring the app is done by editing the source and rebuilding. That seems rude.
  6. If you did update some sort of last-read database, wouldn’t that be a great moment to kick off a background sync? (The Wiki app included in the Zumero SDK does this when a page is created or saved)

Altering a Zumero Table

Mobile offline RSS reader, Part 4

In part 3 of this series, Eric discussed three programs that do the back-end work of RSS aggregation - scanning feed lists, caching summaries and IDs, creating data files as we go.

Before we get into using that information in part 5, we need to adjust one of the schemas a bit.

The items table in each feed’s database is set up as:

1
2
3
4
5
6
7
8
CREATE VIRTUAL TABLE items
USING zumero
(
  id TEXT PRIMARY KEY NOT NULL,
  title TEXT NOT NULL,
  summary TEXT NOT NULL,
  pubdate_unix_time INTEGER NOT NULL
);

Where the id field comes from the feed items’s id or guid element, or from the first link found in the entry. In many RSS feeds, id will also be the permalink to the entry’s “destination”, e.g.

1
2
3
4
<item>
  <title>Enterprise mobile will have a lot of SQL going on</title>
  <guid>http://www.ericsink.com/entries/mobile_sql.html</guid>
  ...

But that’s not mandatory. For example:

1
2
3
4
5
6
<entry>
  <id>http://zumero.com/2013/04/04/just-released-the-zumero-development-server</id>
  <link type="text/html" rel="alternate"
   href="http://zumero.com/2013/04/04/just-released-the-zumero-development-server.html"/>
  <title>Just Released: the Zumero Development Server</title>
  ...

We still want that unique id, but if a link is available, we’ll want to store that, too.

Easy enough when creating a new items table. We’ll alter z_rss_create.cs to read:

1
2
3
4
5
6
7
8
9
10
11
12
db.Execute(
        @"CREATE VIRTUAL TABLE 
        cur.items 
        USING zumero
        (
          id TEXT PRIMARY KEY NOT NULL, 
          title TEXT NOT NULL,
          summary TEXT NOT NULL,
          pubdate_unix_time INTEGER NOT NULL,
          permalink TEXT
        );"
        );

But what about items tables that already exist? Unfortunately, SQLite does not allow alter table ... add column to run against a virtual table like Zumero’s. We’ll use the Zumero-provided alternative, zumero_alter_table_add_column().

In z_rss_update.cs, we’ll see if the new column already exists:

1
2
3
4
5
6
7
8
var query = "pragma cur.table_info(\"items\")";
List<SQLiteConnection.ColumnInfo> cols = db.Query<SQLiteConnection.ColumnInfo> (query);

foreach (var c in cols) {
    found = (string.Compare ("permalink", c.Name, StringComparison.OrdinalIgnoreCase) == 0);
    if (found)
        break;
}

If not…

1
2
3
4
5
6
7
8
9
db.Execute("BEGIN TRANSACTION;");
db.ExecuteScalar<string>(@"select zumero_alter_table_add_column(
   'cur', 'items', 'permalink TEXT');");
db.Execute("COMMIT TRANSACTION;");

// after altering a zumero table, the dbfile connection must be reopened

db.Execute("DETACH cur;", dbfile_name_for_this_feed);
db.Execute("ATTACH ? AS cur;", dbfile_name_for_this_feed);

See this fork of z_rss for all the necessary code.

In part 5, we’ll see a simple iOS RSS reader that pulls these databases in the background, and lets you browse the feed summaries and link through to the full articles.

Veracity 1.5 Released

Just a quick post to note that we’ve released Veracity 1.5.

The most-visible new features:

  • A Tortoise-style extension for Windows users. Jeremy gives you an illustrated once over on the Veracity Q and A site.

Alt text

  • The afore-blogged wiki module is now installed by default, and brings a variety of bug fixes and improvements.

More details to come soon, but I have a race condition to bang my head against.

Meantime, checkout the release notes and downloads.

Building a Veracity Module - Part 4

Finishing up our Veracity module overview, let’s look at a few “fitting in” considerations.

How’d We End Up in the Menu?

If your server_files/ui/modules/yourmodule folder contains a menu.js file, that file will be loaded via a <script> tag in the footer of every Veracity web page.

Whatever you want to add/change in the Veracity menu, do it here. In our case (and most cases), we just append an item to the <ul> named topmenu, which is (surprise!) Veracity’s top-level menu:

1
2
3
4
5
6
7
8
9
var tm = $('#topmenu');

var mi = $("<li id='topmenuwiki'></li>");

var ln = $("<a class='menulink'>wiki</a>").
    attr('href', sgCurrentRepoUrl + "/wiki.html").
    appendTo(mi);

tm.append(mi);

Notice that you can count on jQuery being available to your code.

What About the Activity Stream?

Wiki page changes show up in Veracity’s activity stream, alongside commits, bug updates, etc. The activity stream interface is a simple one: you need to create an object supporting three methods:

  • name(): returns a string describing this particular activity component, for debug logging. Totally up to you.
  • dagsUsed(): returns an array of database DAG IDs, for caching. Include any DAGs your activity stream might query. In our case, it’s [sg.dagnum.WIKI, sg.dagnum.USERS]
  • getActivity(): where all of the work happens

getActivity() returns an array of objects, with (at least) the following members:

  • what: A short description of the object that changed, updated, etc.
  • title: Usually redundant to “what”. Used for Atom entry titles.
  • action: What happened to that thing (created, updated, deleted, fixed…)
  • who: The Veracity user ID to whom this activity should be attributed (a committer, the editor of this particular Wiki change, etc.)
  • when: The (Unix timestamp) time when this activity occurred.
  • link: Optional, a link to the object, its history, etc.

Bug updates, for example, contain (among other things):

1
2
3
4
5
6
7
8
{
    "what": "Work items that reference missing changesets can not be viewed",
    "title: "Work items that reference missing changesets can not be viewed",
    "action": "Fixed X1384",
    "who":"g02d63075631e47bc8a29dad7027f59d382cff0ac413311e0838c60fb42f09aca",
    "when":1313620280696.000000,
    "link":"/workitem.html?recid=gdbb98600a5114533a0a936226f4b2efb8e381b80c91811e0b40f1c6f65d71da9"
}

You should return the most recent N items. The activity stream wrappers will sort them in with other activity sources before returning the JSON or Atom stream.

In the wiki’s case, we build records like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
var record = {
    what: thispage.title,
    title: thispage.title,
    who: thispage.userid,
    when: thispage.timestamp
};

if (first)
    record.action = "Created Wiki page";
else
    record.action = "Edited Wiki page";

if (lastpage)
{
    if (lastpage.title != thispage.title)
    {
        record.action = "Renamed Wiki page";
        record.what += " (was " + lastpage.title + ")";
    }
}

record.link = '/wiki.html?page=' + encodeURIComponent(title);

The first time a page is seen, we report it as “created”; thereafter, as “edited”. If the title changes along the way, we note that instead.

Activity stream including wiki and bug updates

That’s about it, as module high points go. Further questions are very much welcome at the Veracity Q/A site.

Building a Veracity Module - Part 3

So how does the Wiki module work, anyway? Pretty much the way you’d expect a web app to work.

  1. The Wiki form is displayed (and maybe pre-populated with the page’s title and text)
  2. You edit the title and text, and submit
  3. We bundle those fields up into a JSON package
  4. And send that along to the server-side Wiki code
  5. That code either
    1. Updates the page if it already existed, or
    2. Creates a new page
  6. On success, we reload the wiki page

All of the Veracity-specific stuff happens on the server side.

Retrieving a Page

We retrieve the existing page in the GET /wiki/pages/<pagename>.json route (mentioned in Part 2 the other day). Normally (q.v.), that’s as simple as:

1
2
3
4
5
6
7
var db = new zingdb(request.repo, sg.dagnum.WIKI);

var w = vv.where( { "title": request.pagename } );

var recs = db.query('page', ['text','title','recid'], w);

return( recs[0] );

Which translates to “Open the wiki database, find any records matching our pagename, grab their text, title and recid fields, and return the first one.” We can get away with this since our database template requires the pagename to be unique.

A JSON representation of that object is returned (you’ll also see some caching logic in there, but that’s strictly a performance measure, ignorant of the Wiki data).

Creating and Updating

Updates work like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
var csid = newrec._csid || null;
delete newrec._csid;

ztx = db.begin_tx(csid, request.headers.From);
if (newrec.recid)
    rec = ztx.open_record('page', newrec.recid);
else
{
    rec = ztx.new_record('page');
    newrec.recid = rec.recid;
}

rec.title = newrec.title;
rec.text = newrec.text;

vv.txcommit(ztx);
  1. Get the changeset ID (if any) that this update is based on (more on this in a moment)
  2. Start a new transaction
  3. Do we already have a record ID? If so, this is an update. Open that record.
  4. If not, this is a new page. Create a record.
  5. Set the record’s title and text to those passed in from the form.
  6. Commit.

We then return OK to our caller, the page is reloaded, the circle of life continues.

Merges

So what’s up with the changeset ID, and why did we have to say “normally” before?

It’s possible that, by the time you’re saving your changes, someone else has updated the same page. Or maybe your changes are in a nice straight line locally, but a push or pull brings in someone else’s previously-unknown edits. Veracity doesn’t get to throw up its hands and fail. It needs to merge.

And to merge your changes and mine, it needs to know where we each started from. That’s why we pass the changeset IDs around; it tells Veracity “here’s my latest changes, and the last version I knew of was rev 1234”. Later, when Veracity merges that with someone else’s updates, it knows those were based on rev 1235; it finds a common ancestor, does a smart 3-way merge, and all’s well. Almost always.

“Almost always” is not “absolutely always”, though.

What if we both started with:

line 1
line 2

as our text. Then I edited it to read:

line 1
line one and a half
line 2

while off on your machine, you edited it to:

line 1
line 1.5
line 2

Then you pull my changes. Now what? Should your changes be thrown away? Should mine? Should both lines be included? Any of these are possible, but in the template we have to pick one.

The “merge strategy” the Wiki template uses is to concatenate our two texts, and let a human being sort things out. Elsewhere (e.g. in the scrum module) we use all sorts of other strategies, including automatically changing the ID of a work item when it conflicts with one created elsewhere). Since Wiki text is intended for human usage only, and is completely arbitrary, there’s no sense trying to guess the “appropriate” conflict resolution between two edits.

So in this situation, anyone opening the merged page will see:

Wiki page needing manual merge

Edit that as needed, and all’s well with the world again.

Next (and hopefully final) time: plugging into Veracity’s activity stream and cache.

Building a Veracity Module - Part 2

Last time, we installed a Wiki module for Veracity. This time, we’ll look at the pieces that make a module work.

Veracity modules can add to several different parts of the Veracity infrastructure; not every module will touch all of these. They are:

  1. New templated data types (in our case, Wiki pages), and the database DAGs to hold them.
  2. Server-side Javascript code, run in response to either:
    1. Web server requests (creating, updating, viewing Wiki pages), or
    2. Version control hooks (not used in this example)
  3. New Veracity web pages.
  4. Client-side Javascript, run as part of Veracity’s Web UI.

The client-side Javascript is found under server_files/ui/modules/wiki; the rest lives under server_files/modules/wiki. A portion of init.js in that folder bears closer inspection:

1
2
3
4
5
6
7
8
9
area: "wiki",
vendor: sg.vendor.SOURCEGEAR,
grouping: 5,
dagnums: {
    "WIKI": {
        dagid: 1,
        template: 'sg_ztemplate__wiki.json'
    }
},

We’re creating a new database “area” - a group of DAGs that are related to one another. Veracity ships with areas like version_control, and the scrum module defines (not surprisingly) scrum. For modules, the area name should match the containing folder name.

Each module definition needs a vendor ID. Right now, that’s just us. sg.vendor.SOURCEGEAR == 1. If you’re adding your own areas, get in touch, and I’ll make sure you have a vendor ID that doesn’t conflict with anyone else’s.

The grouping property is the number of this area within the vendor’s space. 1-4 were already used (including Scrum), so I added 1. Clever me.

Similarly, dagid is the number of each DAG within this area. If you looked at the scrum module, you’d see the WORK_ITEMS dag has dagid == 1, and the BUILDS dag has dagid == 2.

Every DAG in Veracity needs a template - a description of its record types, their fields, and the merging rules involved for each. All rectypes must be fully, automatically mergeable - failure is, literally, not an option. This allows distributed databases and their owners to remain sane. Merge strategies include “last first”, “greatest”, “uniqify”, etc. We’ll look at the Wiki’s choice next time.

In our init file, we specify the JSON file describing each database template. The templates for core Veracity DAGs can be found in @/src/libraries/templates.

At some magical hand-wavy time that you needn’t worry about, Veracity will look at this init file and:

  1. Make sure the repo we’re playing with has this area installed
  2. Make sure that area has the DAGs we need
  3. Make sure that the DAGs have the right templates set

Once this has happened, the DAGs are available for Javascript use just like the built-in types. If you run the vscript interpreter on a wiki-enabled repo, you can see this:

vscript> sg.to_json__pretty_print( repo.list_areas() )
{
    "core" : 257,
    "version_control" : 258,
    "scrum" : 259,
    "wiki" : 261
}

vscript> sg.to_json__pretty_print(sg.dagnum)
{
    "VERSION_CONTROL" : "0000000010201001",
    "USERS" : "0000000010102062",
    "AREAS" : "0000000010101042",
    "VC_COMMENTS" : "00000000102021c2",
    "VC_TAGS" : "00000000102040c2",
    "VC_BRANCHES" : "0000000010205142",
    "VC_STAMPS" : "00000000102031c2",
    "VC_HOOKS" : "00000000102071c2",
    "TESTING_DB" : "0000000010401002",
    "TESTING2_DB" : "0000000010402002",
    "WORK_ITEMS" : "0000000010301002",
    "BUILDS" : "0000000010302002",
    "WIKI" : "0000000010501002"
}

vscript> db = new zingdb(repo, sg.dagnum.WIKI)
[object zingdb]
vscript> sg.to_json__pretty_print( db.get_template() )
{
    "version" : 1,
    "rectypes" : 
    {
        "page" : 
        {
            "merge" : 
            {
                "merge_type" : "field",
            // ...
            }
        // ...
        }
    }
    // ...
}

And after creating a page or two:

vscript> db = new zingdb(repo, sg.dagnum.WIKI)
vscript> records = db.query('page', ['*'])
[object Object],[object Object]
vscript> sg.to_json__pretty_print(records[1])
{
    "recid" : "ge9dbadde62004708abd960d58a99753f191e0f24c42b11e0a1a0c8bcc8e13b9a",
    "text" : "This is, in fact, another page entirely.  \n\n[[Yet Another Page]]",
    "title" : "Another Page"
}

Veracity will also install any URIs added by the module’s server-side Javascript. Look at server_files/modules/wiki/wiki.js to see how those are specified:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
"/repos/<repoName>/wiki/page/<pagename>.json": {
    "GET": {
        onDispatch: function (request) {
           // this handles the request for a Wiki page's current contents
           // ...
       }
    }
},

"/repos/<repoName>/wiki/page": {
    "POST": {
        onJsonReceived: function (request, newrec) {
           // here we receive JSON (in newrec) describing a page to be updated or created
           // ...
        }
     }
}

These are used in Ajax calls from server_files/ui/modules/wiki/wiki.js. For example, when rendering links to other wiki pages, we validate those links by attempting to retrieve them:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
var purl = sgCurrentRepoUrl + "/wiki/page/" + pageName + ".json";

vCore.ajax(
{
    url: purl,
    dataType: 'json',
    reportErrors: false,
    success: function(data) {
        vvWiki.setGoodPage(ln, pageName);
    },
    error: function() {
        vvWiki.setBadPage(ln, pageName);
    }
});

Next up: we follow the code through the creation, update, and merge of a Wiki page.

Building a Veracity Module - Part 1

Veracity was built to be extensible. Not just by virtue of the source being available (although that helps), but also via the embedded JavaScript interpreter. Without writing a line of C, new HTML pages and REST urls can be added, supporting entirely new data types and functionality.

The “native” Scrum functions in Veracity (work items, build tracking, time tracking, milestones, filters, etc.) are nowhere to be found in the Veracity library code. It’s Javascript all the way down, helped out by client-side scripts and HTML. As we add new (or alternative) features ourselves, expect to see many of them show up as new “modules” of pluggable (and un-pluggable) code.

We were going to be adding a Wiki module to Veracity eventually, anyway; and since we wanted to show you how modules are built, a simple Wiki seemed like a nice place to start. This isn’t production code (yet), but it’s an instructive start.

For today, let’s just get the Wiki installed in your Veracity server and make sure everything’s up and running.

Whether you’re building from source or running from a pre-built installer, you’ll need to download the latest nightly build (1.0.1.10527 or later) to retrieve the Wiki module. If you’ve cloned from our public repository, you’ll want to pull the latest.

Within your source folder or the unpacked archive, look for .../src/modules/wiki. Within that directory, you’ll see a README file, some license material, a test directory, and the part you actually care about: a server_files folder.

You’ll want to copy the contents of that folder into Veracity’s server files folder. If you’re unsure where that is, run vv config. Towards the end, you’ll see something like:

server/files: /home/alanswann/veracity/src/server_files

in this case, from within modules/wiki, you’d say (on Unix-y systems):

[~/veracity/src/modules/wiki]
$ cp -R server_files/* /home/alanswann/veracity/src/server_files

and your Wiki code should now included with the rest of Veracity.

[~/veracity/src/modules/wiki]
$ ls ~/veracity/src/server_files/modules
scrum  wiki

Note the above-mentioned scrum module alongside the wiki.

Start (or restart) vv serve, and you should see a wiki item in the top menu.

Veracity menu with wiki

Click that. You’ll see a default home page, explaining that, hey, it’s a default home page. You can edit this page (using Markdown), or click the new page link to add another page.

As pages are created, they’ll show up in the right sidebar, to be clicked and viewed, or inserted into another page’s edit box as links.

Editing a Wiki entry

Next post, we’ll start looking at the code. The good news? All of the heavy lifting is done for us, from the flexibly-licensed editor and preview tools, to the Veracity libraries that handle saves, updates and merges (gotta have merges, it’s a distributed Wiki, after all). I just glued them together.