Just a quick post to note that we’ve released Veracity 1.5.

The most-visible new features:

• A Tortoise-style extension for Windows users. Jeremy gives you an illustrated once over on the Veracity Q and A site.

• The afore-blogged wiki module is now installed by default, and brings a variety of bug fixes and improvements.

More details to come soon, but I have a race condition to bang my head against.

Meantime, checkout the release notes and downloads.

Finishing up our Veracity module overview, let’s look at a few “fitting in” considerations.

How’d We End Up in the Menu?

If your server_files/ui/modules/yourmodule folder contains a menu.js file, that file will be loaded via a <script> tag in the footer of every Veracity web page.

Whatever you want to add/change in the Veracity menu, do it here. In our case (and most cases), we just append an item to the <ul> named topmenu, which is (surprise!) Veracity’s top-level menu:

var tm = $('#topmenu');

var mi = $("<li id='topmenuwiki'></li>");

var ln = $("<a class='menulink'>wiki</a>").
    attr('href', sgCurrentRepoUrl + "/wiki.html").
    appendTo(mi);

tm.append(mi);

Notice that you can count on jQuery being available to your code.

What About the Activity Stream?

Wiki page changes show up in Veracity’s activity stream, alongside commits, bug updates, etc. The activity stream interface is a simple one: you need to create an object supporting three methods:

• name(): returns a string describing this particular activity component, for debug logging. Totally up to you.
• dagsUsed(): returns an array of database DAG IDs, for caching. Include any DAGs your activity stream might query. In our case, it’s [sg.dagnum.WIKI, sg.dagnum.USERS]
• getActivity(): where all of the work happens

getActivity() returns an array of objects, with (at least) the following members:

• what: A short description of the object that changed, updated, etc.
• title: Usually redundant to “what”. Used for Atom entry titles.
• action: What happened to that thing (created, updated, deleted, fixed…)
• who: The Veracity user ID to whom this activity should be attributed (a committer, the editor of this particular Wiki change, etc.)
• when: The (Unix timestamp) time when this activity occurred.
• link: Optional, a link to the object, its history, etc.

Bug updates, for example, contain (among other things):

{
    "what": "Work items that reference missing changesets can not be viewed",
    "title: "Work items that reference missing changesets can not be viewed",
"action": "Fixed X1384",
    "who":"g02d63075631e47bc8a29dad7027f59d382cff0ac413311e0838c60fb42f09aca",
    "when":1313620280696.000000,
    "link":"/workitem.html?recid=gdbb98600a5114533a0a936226f4b2efb8e381b80c91811e0b40f1c6f65d71da9"
}

You should return the most recent N items. The activity stream wrappers will sort them in with other activity sources before returning the JSON or Atom stream.

In the wiki’s case, we build records like so:

var record = {
    what: thispage.title,
    title: thispage.title,
    who: thispage.userid,
    when: thispage.timestamp
};

if (first)
    record.action = "Created Wiki page";
else
    record.action = "Edited Wiki page";

if (lastpage)
{
    if (lastpage.title != thispage.title)
    {
        record.action = "Renamed Wiki page";
        record.what += " (was " + lastpage.title + ")";
    }
}

record.link = '/wiki.html?page=' + encodeURIComponent(title);

The first time a page is seen, we report it as “created”; thereafter, as “edited”. If the title changes along the way, we note that instead.

That’s about it, as module high points go. Further questions are very much welcome at the Veracity Q/A site.

So how does the Wiki module work, anyway? Pretty much the way you’d expect a web app to work.

1. The Wiki form is displayed (and maybe pre-populated with the page’s title and text)
2. You edit the title and text, and submit
3. We bundle those fields up into a JSON package
4. And send that along to the server-side Wiki code
5. That code either
   1. Updates the page if it already existed, or
   2. Creates a new page
6. On success, we reload the wiki page

All of the Veracity-specific stuff happens on the server side.

Retrieving a Page

We retrieve the existing page in the GET /wiki/pages/<pagename>.json route (mentioned in Part 2 the other day). Normally (q.v.), that’s as simple as:

var db = new zingdb(request.repo, sg.dagnum.WIKI);

var w = vv.where( { "title": request.pagename } );

var recs = db.query('page', ['text','title','recid'], w);

return( recs[0] );

Which translates to “Open the wiki database, find any records matching our pagename, grab their text, title and recid fields, and return the first one.” We can get away with this since our database template requires the pagename to be unique.

A JSON representation of that object is returned (you’ll also see some caching logic in there, but that’s strictly a performance measure, ignorant of the Wiki data).

Creating and Updating

Updates work like so:

var csid = newrec._csid || null;
delete newrec._csid;

ztx = db.begin_tx(csid, request.headers.From);
if (newrec.recid)
    rec = ztx.open_record('page', newrec.recid);
else
{
    rec = ztx.new_record('page');
    newrec.recid = rec.recid;
}

rec.title = newrec.title;
rec.text = newrec.text;

vv.txcommit(ztx);

1. Get the changeset ID (if any) that this update is based on (more on this in a moment)
2. Start a new transaction
3. Do we already have a record ID? If so, this is an update. Open that record.
4. If not, this is a new page. Create a record.
5. Set the record’s title and text to those passed in from the form.
6. Commit.

We then return OK to our caller, the page is reloaded, the circle of life continues.

Merges

So what’s up with the changeset ID, and why did we have to say “normally” before?

It’s possible that, by the time you’re saving your changes, someone else has updated the same page. Or maybe your changes are in a nice straight line locally, but a push or pull brings in someone else’s previously-unknown edits. Veracity doesn’t get to throw up its hands and fail. It needs to merge.

And to merge your changes and mine, it needs to know where we each started from. That’s why we pass the changeset IDs around; it tells Veracity “here’s my latest changes, and the last version I knew of was rev 1234”. Later, when Veracity merges that with someone else’s updates, it knows those were based on rev 1235; it finds a common ancestor, does a smart 3-way merge, and all’s well. Almost always.

“Almost always” is not “absolutely always”, though.

What if we both started with:

line 1
line 2

as our text. Then I edited it to read:

line 1
line one and a half
line 2

while off on your machine, you edited it to:

line 1
line 1.5
line 2

Then you pull my changes. Now what? Should your changes be thrown away? Should mine? Should both lines be included? Any of these are possible, but in the template we have to pick one.

The “merge strategy” the Wiki template uses is to concatenate our two texts, and let a human being sort things out. Elsewhere (e.g. in the scrum module) we use all sorts of other strategies, including automatically changing the ID of a work item when it conflicts with one created elsewhere). Since Wiki text is intended for human usage only, and is completely arbitrary, there’s no sense trying to guess the “appropriate” conflict resolution between two edits.

So in this situation, anyone opening the merged page will see:

Edit that as needed, and all’s well with the world again.

Next (and hopefully final) time: plugging into Veracity’s activity stream and cache.

Last time, we installed a Wiki module for Veracity. This time, we’ll look at the pieces that make a module work.

Veracity modules can add to several different parts of the Veracity infrastructure; not every module will touch all of these. They are:

1. New templated data types (in our case, Wiki pages), and the database DAGs to hold them.
2. Server-side Javascript code, run in response to either:
   1. Web server requests (creating, updating, viewing Wiki pages), or
   2. Version control hooks (not used in this example)
3. New Veracity web pages.
4. Client-side Javascript, run as part of Veracity’s Web UI.

The client-side Javascript is found under server_files/ui/modules/wiki; the rest lives under server_files/modules/wiki. A portion of init.js in that folder bears closer inspection:

area: "wiki",
vendor: sg.vendor.SOURCEGEAR,
grouping: 5,
dagnums: {
    "WIKI": {
        dagid: 1,
        template: 'sg_ztemplate__wiki.json'
    }
},

We’re creating a new database “area” - a group of DAGs that are related to one another. Veracity ships with areas like version_control, and the scrum module defines (not surprisingly) scrum. For modules, the area name should match the containing folder name.

Each module definition needs a vendor ID. Right now, that’s just us. sg.vendor.SOURCEGEAR == 1. If you’re adding your own areas, get in touch, and I’ll make sure you have a vendor ID that doesn’t conflict with anyone else’s.

The grouping property is the number of this area within the vendor’s space. 1-4 were already used (including Scrum), so I added 1. Clever me.

Similarly, dagid is the number of each DAG within this area. If you looked at the scrum module, you’d see the WORK_ITEMS dag has dagid == 1, and the BUILDS dag has dagid == 2.

Every DAG in Veracity needs a template - a description of its record types, their fields, and the merging rules involved for each. All rectypes must be fully, automatically mergeable - failure is, literally, not an option. This allows distributed databases and their owners to remain sane. Merge strategies include “last first”, “greatest”, “uniqify”, etc. We’ll look at the Wiki’s choice next time.

In our init file, we specify the JSON file describing each database template. The templates for core Veracity DAGs can be found in @/src/libraries/templates.

At some magical hand-wavy time that you needn’t worry about, Veracity will look at this init file and:

1. Make sure the repo we’re playing with has this area installed
2. Make sure that area has the DAGs we need
3. Make sure that the DAGs have the right templates set

Once this has happened, the DAGs are available for Javascript use just like the built-in types. If you run the vscript interpreter on a wiki-enabled repo, you can see this:

vscript> sg.to_json__pretty_print( repo.list_areas() )
{
    "core" : 257,
    "version_control" : 258,
    "scrum" : 259,
    "wiki" : 261
}

vscript> sg.to_json__pretty_print(sg.dagnum)
{
    "VERSION_CONTROL" : "0000000010201001",
    "USERS" : "0000000010102062",
    "AREAS" : "0000000010101042",
    "VC_COMMENTS" : "00000000102021c2",
    "VC_TAGS" : "00000000102040c2",
    "VC_BRANCHES" : "0000000010205142",
    "VC_STAMPS" : "00000000102031c2",
    "VC_HOOKS" : "00000000102071c2",
    "TESTING_DB" : "0000000010401002",
    "TESTING2_DB" : "0000000010402002",
    "WORK_ITEMS" : "0000000010301002",
    "BUILDS" : "0000000010302002",
    "WIKI" : "0000000010501002"
}

vscript> db = new zingdb(repo, sg.dagnum.WIKI)
[object zingdb]
vscript> sg.to_json__pretty_print( db.get_template() )
{
    "version" : 1,
    "rectypes" : 
    {
        "page" : 
        {
            "merge" : 
            {
                "merge_type" : "field",
            // ...
            }
        // ...
        }
    }
    // ...
}

And after creating a page or two:

vscript> db = new zingdb(repo, sg.dagnum.WIKI)
vscript> records = db.query('page', ['*'])
[object Object],[object Object]
vscript> sg.to_json__pretty_print(records[1])
{
    "recid" : "ge9dbadde62004708abd960d58a99753f191e0f24c42b11e0a1a0c8bcc8e13b9a",
    "text" : "This is, in fact, another page entirely.  \n\n[[Yet Another Page]]",
    "title" : "Another Page"
}

Veracity will also install any URIs added by the module’s server-side Javascript. Look at server_files/modules/wiki/wiki.js to see how those are specified:

"/repos/<repoName>/wiki/page/<pagename>.json": {
    "GET": {
        onDispatch: function (request) {
           // this handles the request for a Wiki page's current contents
           // ...
       }
    }
},

"/repos/<repoName>/wiki/page": {
    "POST": {
        onJsonReceived: function (request, newrec) {
           // here we receive JSON (in newrec) describing a page to be updated or created
           // ...
        }
     } 
}        

These are used in Ajax calls from server_files/ui/modules/wiki/wiki.js. For example, when rendering links to other wiki pages, we validate those links by attempting to retrieve them:

var purl = sgCurrentRepoUrl + "/wiki/page/" + pageName + ".json";

vCore.ajax(
{
    url: purl,
    dataType: 'json',
    reportErrors: false,
    success: function(data) {
        vvWiki.setGoodPage(ln, pageName);
    },
    error: function() {
        vvWiki.setBadPage(ln, pageName);
    }
});

Next up: we follow the code through the creation, update, and merge of a Wiki page.

Veracity was built to be extensible. Not just by virtue of the source being available (although that helps), but also via the embedded JavaScript interpreter. Without writing a line of C, new HTML pages and REST urls can be added, supporting entirely new data types and functionality.

The “native” Scrum functions in Veracity (work items, build tracking, time tracking, milestones, filters, etc.) are nowhere to be found in the Veracity library code. It’s Javascript all the way down, helped out by client-side scripts and HTML. As we add new (or alternative) features ourselves, expect to see many of them show up as new “modules” of pluggable (and un-pluggable) code.

We were going to be adding a Wiki module to Veracity eventually, anyway; and since we wanted to show you how modules are built, a simple Wiki seemed like a nice place to start. This isn’t production code (yet), but it’s an instructive start.

For today, let’s just get the Wiki installed in your Veracity server and make sure everything’s up and running.

Whether you’re building from source or running from a pre-built installer, you’ll need to download the latest nightly build (1.0.1.10527 or later) to retrieve the Wiki module. If you’ve cloned from our public repository, you’ll want to pull the latest.

Within your source folder or the unpacked archive, look for .../src/modules/wiki. Within that directory, you’ll see a README file, some license material, a test directory, and the part you actually care about: a server_files folder.

You’ll want to copy the contents of that folder into Veracity’s server files folder. If you’re unsure where that is, run vv config. Towards the end, you’ll see something like:

server/files: /home/alanswann/veracity/src/server_files

in this case, from within modules/wiki, you’d say (on Unix-y systems):

[~/veracity/src/modules/wiki]
$ cp -R server_files/* /home/alanswann/veracity/src/server_files

and your Wiki code should now included with the rest of Veracity.

[~/veracity/src/modules/wiki]
$ ls ~/veracity/src/server_files/modules
scrum  wiki

Note the above-mentioned scrum module alongside the wiki.

Start (or restart) vv serve, and you should see a wiki item in the top menu.

Click that. You’ll see a default home page, explaining that, hey, it’s a default home page. You can edit this page (using Markdown), or click the new page link to add another page.

As pages are created, they’ll show up in the right sidebar, to be clicked and viewed, or inserted into another page’s edit box as links.

Next post, we’ll start looking at the code. The good news? All of the heavy lifting is done for us, from the flexibly-licensed editor and preview tools, to the Veracity libraries that handle saves, updates and merges (gotta have merges, it’s a distributed Wiki, after all). I just glued them together.

This may well be obvious to anyone bothering to read this blog, but it’s helpful enough for me that I thought I’d scribble it down, anyway.

You’ve made a minor change, just adding some validation code:

$ vv status
Modified:  @/validate.c

$ vv diff
=== ================
===   Modified: File @/validate.c
--- @/validate.c    4b8c0a0278cc18fdeb8592a6b56b81ba4c4b6841
+++ @/validate.c    2011/07/11 19:41:01.000 +0000
@@ -4,4 +4,5 @@
 void validate(const char *user)
 {
         printf("%s, you're awesome.\n", user);
+        printf("And attractive.\n");
 }

You commit, then pull the latest from your team before pushing your changes up.

There are inded some changes, so you’ll need to merge:

$ vv heads

    revision:  7:11e7676e2d8c96071d6ae6748afb29fbea291d3c
      branch:  master
         who:  otherguy@example.com
        when:  2011/07/11 15:44:24.448 -0400
     comment:  more changes you don't care about
      parent:  5:54675c9beeab003fce135282654cf36f9032f326

    revision:  6:3926e0614eb2164bb0839eb1c6ba2c4954107dcf
      branch:  master
         who:  me@example.com
        when:  2011/07/11 15:42:08.610 -0400
     comment:  additional validation
      parent:  4:55205503daa35db9fd3699473da84f49493ef03c

$ vv merge
4 updated, 0 deleted, 3 added, 0 merged, 0 unresolved

$ vv status
   Added:  @/othercode.c
   Added:  @/othercode.h
   Added:  @/whatisthisidonteven.c
Modified:  @/addsprint.js
Modified:  @/connect.js
Modified:  @/reqtest.js
Modified:  @/validate.h

Hrm. A lot of code, none of it yours. The resulting diff is so long I won’t bother to fake it up for the blog. But are you sure none of it’s yours? Skimming through hundreds of lines of diff output looking for problem code is not fun. And all for your little one-liner.

So turn the merge on its head. Starting from the other side, you should easily be able to tell if your changes are being merged in a sane fashion.

# clean slate
$ vv revert --all

# start from the other guy's changes
$ vv update -r 7

# merge in yours
$ vv merge
1 updated, 0 deleted, 0 added, 0 merged, 0 unresolved

$ vv status
Modified:  @/validate.c

$ vv diff
=== ================
===   Modified: File @/validate.c
--- @/validate.c        4b8c0a0278cc18fdeb8592a6b56b81ba4c4b6841
+++ @/validate.c        2011/07/11 19:46:21.000 +0000
@@ -4,4 +4,5 @@
 void validate(const char *user)
 {
        printf("%s, you're awesome.\n", user);
+       printf("And attractive.\n");
 }

# looks familiar. we're good.
$ vv commit -m"merge"

Chipping away at my Veracity tasks today, I ran across an issue that’s common in DVCS + Centralized Bug Tracking scenarios, but unnecessary and easily-avoided in Veracity.

In a nutshell, it’s this:

1. I commit some changes, locally, to fix a bug.
2. I mark the bug fixed, on our one and only bug tracker instance.
3. I don’t push my code just yet.

Until that code is pushed to the shared server, QA will have my bug in their to-verify list, but won’t have the code to do so.

I might delay the push for a number of reasons - waiting to finish a couple of related bugs and push as a unit; needing to merge and test someone else’s latest code before pushing the results back; etc. Until I’m done, though, we’re out of sync.

You can keep track via special statuses, tags, stamps, etc. (and remember to reset them once you’ve pushed), but why?

When I’m working on Veracity, I’m usually working against a local instance of the bug tracking / web UI. It’s faster, I’m not competing for resources, and I get to do things like associate commits to bugs right from the command line while I work.

It also avoids the problem above, since my workflow looks like this:

This is normal in the Veracity world, and unremarkable except in those instances, like this afternoon, when someone marks a bug closed without pushing up the associated code. Not naming names. We all make mistakes, and I’m sure writing a book can be pretty distracting.

I’m a developer at SourceGear, working on Veracity these days.

Elsewhere in my life, there’s mainly music - you can find out more about that at paulroub.com.

My complaining-about-air-travel needs are handled on Twitter as @paulroub.

So far, most of the discussions about Veracity have been about its DVCS abilities. I thought I’d talk a bit about the other side of Veracity, the bug/work tracking side, from the users’ point of view. I’ll start with the Activity Stream.

One choice we made for the initial releases was to forego email notifications as a means of alerting users to new work, updated bug reports, code changes, and so on. There were a number of reasons for this:

• Since the work item activity is distributed, bugs may be added and updated from multiple machines, sometimes behind-the-scenes, sometimes offline (I’m sitting in an airport right now, waiting to act out the canonical “dev on a plane” DVCS scenario). When should email be sent? By whom?
• Cross-platform email support would take time away from features we really want in the first release.
• We all get enough email as it is.
• There are less-irritating ways to keep people in the loop.

Of course, Veracity has really nice search and filter tools, allowing you to see bugs assigned to you; bugs you need to verify; what’s up for you in the next sprint; any bug mentioning “snickers” or stamped with “SOOPER URGENT”; etc.

But we also wanted to give you a view of what’s going on with your project in a more ambient way, so we added the Activity Stream. Every 30 seconds, if anything new of interest has happened in your repository, it quietly updates to let you know.

What’s In There?

The activity stream shows you the most-recent 25 things you might care about. In particular, you’ll see:

• Newly-created work items
• Modified work items (status changes, comments, relations, etc.)
• Version control commits
• Build failures (if your build system is using our reporting scripts)
• And perhaps more, by the time you read this

Here’s a shot of the last few items in dev@example.com’s activity stream:

What’s Not?

The goal of the activity stream is to be useful, as opposed to complete. If you look at the activity stream side-by-side with our version control history graph, you’ll notice that rev #7 isn’t listed, even though it’s the latest. Why not? Its comment was simply “merge”. That’s not terribly interesting, you probably don’t care, so we hide it. We also hide no-op work item updates, successful builds (our continuous integration setup spins those out constantly, and they’d push the more-urgent stuff off screen), etc.

The “my updates only” option

You can also narrow down the work items shown in the stream to just those that you’re involved with. By checking “my updates only”:

you’ll only see work items where your username is either the Reporter, Assignee, or Verifier, or where the item has been stamped with your username.

You’ll still see all commits, since other people’s commits are certainly likely to be of interest to you. You’ll also still see your own commits, which are not likely to surprise you, so that you can drag them around.

Dragging From the Activity Stream

Veracity work items can be related to each other (this depends on that, this is a duplicate of the other, etc.). They can also be associated with version control commits – check in some changes, mark the relevant bug.

All that is nicely searchable in the work item edit page, but when you want to associate with another recent bug, or with a recent commit, why type anything? You’re already looking at that item in the stream, so just drag it into the appropriate slot and Save.

Your Stream in Your Feed Reader

For several of us on the Veracity team, the first we hear of a newly-assigned or reopened bug is when it shows up in our feed reader of choice. The Veracity activity streams for any repository (either the all-inclusive or my-updates-only version) are also served as Atom feeds. The link for whichever version of the stream you’re viewing is always available via the feed icon at the bottom of the stream:

Here’s this dev’s full-blown stream:

and his mine-mine-mine stream:

Note that the titles are live links to the relevant Veracity work item, changeset detail, etc. pages.

Keep In Touch

As ever, feedback, questions, and suggestions are very much welcome on the Veracity users mailing list. Drop in.

We’ll also have a booth at OSCON again this July, so stop by, say “hi”, kick the tires, argue about rebase…

I think, with this post, I’ll have reached a new peak in my quest to share scripts that almost no one else will use. What is the overlap between “users of the Todo.txt Touch Android app” and “pre-release Veracity users”? The odds of that being “just me” are favorable. I expect those odds to change. In the meantime, I’ve already written the script; it would be rude not to share.

Veracity, if you’ve been paying attention, is Sourcegear’s upcoming Distributed Version Control / Bug Tracking / more system with an open sourced core. The main reason this blog’s been so dead? “Busy. Busy coding. Can’t talk right now.” That kinda thing.

todo.txt is Gina Trapani’s script/app that aims to make simple todo.txt files more useful and mobile, while still maintaining the simplicity we love. I’m a fan and a daily user of the command-line script and the Android app.

One thing Veracity doesn’t have, as yet, is a mobile interface. Since all the boxes on which I deal with Veracity are behind firewalls, I couldn’t use one right now, anyway. But that doesn’t stop me from realizing I need to log myself a task or two when I’m out and away from the keyboard. What to do?

One of the thing’s Gina’s Android app does is sync that todo.txt file via Dropbox. I add a todo item, and in a few seconds, it’s also in todo.txt on my laptop, where an instance of the Veracity team’s repository resides. Also, we have a nice little library of Javascript functions (on which much of Veracity’s web API is built), all callable from our vscript command-line JS tool. See where this is going?

I add a new item in the todo.txt app, like so:

See how I set a “context” of @veracity? That will matter later. If I wished, I could also add tags (things prefixed by “+” in todo.txt’s world), and set a priority from (A) to (E). Here’s a few of those, after the auto-sync:

I have a cron job that runs every 20 minutes, along the lines of:

0,20,40 * * * * vscript todo2vv.js /path/of/todo.txt paul.roub@sourcegear.com veracity-pr

Which tells the the todo2vv.js script (download) to look in todo.txt for any @veracity items, mark them done in the todo.txt list, and add them instead to my Veracity todo list in the veracity-pr repository. Here they are after import:

And filtered by priority:

All have been assigned to our current milestone-in-progress (Sprint 18, the Veracity team being a scrummy one), stamped “todo” (along with any of those todo.txt tags), and given priorities matching any seen in the text file.

The script is fairly simple; it spends more time parsing todo.txt and managing files than creating the actual work items. Feedback, questions, comments are most welcome; at the moment, they’re probably best asked on the veracity-users mailing list.

Back to work. Gotta change the activity stream sorting.