When I first started developing, I wrote in a loop timer that keeps track of the longest processing time as well as the average loop time. After the introduction of a large (16,200 room) world map, the loop times jumped from 10-30 µsec to over 190,000 µsec—no bueno.

So last night I ran Valgrind and used kcachegrind to view the results, which showed me this:

Put simply, most of my time is spent manipulating mutexes. I'm going to have to rethink my design, because that's way too much time…

I've made a few modifications lately, the biggest being changing the command objects to Singleton objects since that's the way I've been using them anyway. I've also been working on the get command, and as happens when a design gets better, it's shrinking in size. Hopefully I'll be able to wrap it up tomorrow, since Lorien will be working until almost 9:30pm tomorrow night anyway.

I've also been lax about source control. My current system is rolling up a tarball, bz2-compressing it, and dropping it into my gmail account. I really should get something a little more sophisticated. At some point I'm going to have to do something about it. But the goal right now is to get it building again, and to test out the get revision.

As I was making some adjustments to my I/O last night I ran across a bit of documentation that I never noticed before. Of course, it could be newly added. Anyway, I've found the most convenient way to parse my documents (I have many thousands of them to read and write), is to access them by node-name. For example:

node["location"]["name"] >> mObjectLocation.location;

The information I read last night, however, may convince me to change this behavior.

Apparently, named-node access in yaml-cpp is order-n² complex over the entire document. Put simply, every time you access a node like this the library loops through all nodes looking for a match. Personally, I think a hashed table lookup would have been much faster, and perhaps they'll add that later.

Overall, I wouldn't normally worry about it because my documents tend to only have a few dozen nodes. But I do have 20,000 documents to parse. And to top that off, as they are used they become more complex. And as more items are added, the node size will grow geometrically. I can easily envision a future where a single document contains hundreds of nodes, and it's starting to scare me.

The solution? Well, I can honestly say I don't have a good one yet. I think perhaps if I enforce the order things are written, which already only happens in one way, I can then read in linearly instead of by node-name. The problem with that is if a file is edited by hand and the order is accidentally changed. So by renaming my current solution to readCompatible() and writing a new linear read() function, I should cut down on a lot of extra unnecessary processing.

But I'm still not sure that's the best solution. Any smart people out there have a better idea?

Unfortunately, I still can't seem to get qi::phrase_parse to work right. The problem is that I'm trying to match one of a set of increasingly complicated grammar structures (perhaps starting at the most complicated and working backwards would fix this issue…). So, here is my problem in gruesome detail:

I'm implementing a get command for a MUD, an online text-only game system. Since I'm writing it from scratch, I decided to use some of the most up-to-date tools I could find to do so, and Boost has some terrific libraries that make this possible. As far as I can discern, the get command can come in the following forms:

1. get hammer
2. get 20 nails
3. get hammer 2 (the second hammer)
4. get 20 nails 3 (20 of the 3rd type of nails visible)
5. get hammer from toolbelt
6. get 20 nails from bucket
7. get hammer from toolbox 2 (the hammer from the second toolbox)
8. get 20 nails from bucket 3 (you get the idea)
9. get hammer 3 from toolbox 2
10. get 20 nails 2 from bucket 3

The problem comes when using phrase_parse with an empty space skip-parser. Given the two rules:

bool res = qi::phrase_parse(first, str.end(),
				(
					+qi::alpha
				),
				' ',
				item
);

And

bool res = qi::phrase_parse(first, str.end(),
					(
						+qi::alpha
						>> "from"
						>> +qi::alpha
					),
					' ',
					item,
					container
);

If I check the phrase get hammer from bucket it matches rule one, because after you remove all the spaces from the string it's just one long word! This is an unexpected result, and even if I check rule two before rule one I only match rule one!

And now I'm really lost. I imagine there's a better way to do this, but I can't seem to figure it out. Also, since my allergies are going crazy, I've got to take the night off anyway. Anybody out there know of a solution?

For example, in Spirit classic, I had this rule to handle the most complex rule for my get command. In this case, it handles a command like get 20 items 2 from container 3:

rule6 = int_p[assign_a(mNumberToGet)] >> ' ' >>
			(+alpha_p)[assign_a(mItemToFind)] >> ' ' >>
			int_p[assign_a(mItemNumber)] >> ' ' >>
			str_p("from") >> ' ' >>
			(+alpha_p)[assign_a(mContainerToLookIn)] >> ' ' >>
			int_p[assign_a(mContainerNumber)];

which could be tested by the following code: if(parse(str.c_str(), rule6).full)

Here is the equivalent code in Spirit 2

	std::string::iterator first = str.begin();

	bool r = qi::parse(first, str.end(),
					(
						qi::int_ >> ' '
						>> +(~qi::char_(' ')) >> ' '
						>> qi::int_ 	>> ' '
						>> qi::lit("from") >> ' '
						>> +(~qi::char_(' ')) >> ' '
						>> qi::int_
					),
					mNumberToGet,
					mItemToFind,
					mItemNumber,
					mContainerToLookIn,
					mContainerNumber
	);

	if(r && first == command.end()) {
		return true;
	}

In many ways, the new Spirit 2 library is better, and purports to be much faster as well. The parse function has certainly been made somewhat more convenient. Overall, I'd rather not have had to spend time on this, but the only other option is to lock in with an old version of Boost that may hamper me in the future.

There are nine different ways to use the get command, each with its own grammar rule. Each of the rules gets more programmatically complicated, so the command code itself got pretty long.

I do suspect that the get and put functions will be some of the most complicated to write. I'd say I'm halfway done, but I haven't done enough testing on the get code to really call it "done."

• get book
• get book 2
• get 30 coins
• get book from shelf
• get book 3 from shelf
• get book from shelf 2
• get book 2 from shelf 3
• get 30 coins from bag 2
• get 5 arrows 2 from quiver 3

I had to define a search order, too. If a player is carrying a container, and a similarly-named container also exists in the player's current room, the player can only manipulate the one in their inventory, and would have to first pick up container, then take items from it. I could work out a way to combine the inventories of the player and their location, but I'm not sure I want to go that far. Honestly, the way I've coded things it's not too difficult, but I'm more worried about perception of items and the confusion combining them all could potentially cause.

In any case, the get command is growing in complexity by orders of magnitude right now, and it's almost too much to keep in my head. I'm going to have to go through and refactor it before I've even finished writing it.

Here's a code example of one of my rules:

// 'get 20 items from container 2' grammar
rule5a = boost::spirit::int_p[boost::spirit::assign_a(mNumberToGet)] >> ' ' >>
	(+boost::spirit::alpha_p)[boost::spirit::assign_a(mItemToFind)] >> ' ' >>
	boost::spirit::str_p("from") >> ' ' >>
	(+boost::spirit::alpha_p)[boost::spirit::assign_a(mContainerToLookIn)] >> ' ' >>
	boost::spirit::int_p[boost::spirit::assign_a(mContainerNumber)];

Which reminds me, since this class (the get command) is a shared pointer, I have to lock up the volatile bits in a mutex so multiple threads play nice together.

I know I'm on the right track, because in nearly every case, the code changes were smaller than the original code. So far, that's always been a good sign for me.

The best part is now I have loadable and saveable objects all working, and I can continue to build the rest of the functionality to complete the MUD and make it playable. Thank goodness for lazy vacations!

On with the update! If you keep up with this blog (and I can only assume you do if you read this), you know I've run in to issues with unstructured text files and saving complicated data objects to a storage media and then restoring them. I began a search for the right solution—a white whale, a magic bullet, whatever you wish to call it. My first instinct was to leap to XML because it can surely solve this small problem with ease. Right?

Wrong! XML is complicated, both to read and write (by machines anyway). Yes, there are libraries to help with that. Oh there are such libraries that you have no idea what you may be getting yourself in to (most of you, anyway).

Throwing out XML, and going with a cow-orker's recommendation of implementing JSON, I began writing a proof-of-concept program to save and restore data, but quickly ran in to a serious issue: JSON doesn't support multi-line strings.

Yet more searching around yielded a link to YAML, which I posted about the other day. YAML is easy to read from a human standpoint, and even has a couple of C++ libraries to help read it. YAML supports everything I want to do. So I scrapped the JSON project and started a proof-of-concept for YAML. I quickly ran in to an issue reading YAML, but it turned out to be my own fault: I didn't understand how a part of the library worked. Shortly after that, though, I did uncover a bug in the yaml-cpp library with indicator characters used as scalars (this is where you non-technical people either stop reading, or stick your fingers in your ears—metaphorically—and ignore me from here on out). With that issue wrapped up, serious work began and the proof-of-concept grew wings and took off. Just like I hoped it would.

Now I'm in the midst of tearing out underlying serialization code (the part that handles the loading and saving of objects), and it looks like it may take many hours to get things back up and working again. But hopefully by that time I'll have a full-blown, ready-to-use system that will require very little future modification.

Since then I have also become aware of (thanks Keith) the Boost Serialization library. It does the same thing I'm looking at doing, but a little more generically, just like Boost always does. Now the dilemma is: do I go forward with the YAML code, or switch to boost::serialization? Perhaps another proof-of-concept program should be written to determine this, but I need to do a little more research. Right now I have no idea how the serialized data looks to human eyes, and I haven't found any examples yet. It's important to my project to have these files as easily modified by humans as by machines.

In summary, that's where the project sits today. I may have some time this week to work on it, but if it's anything like last week I'm not going to have the brainwidth (I just came up with that word!) to work on it. Fortunately, Lorien and I plan a vacation to see family in Oregon in another week, so I should have a few idle days to work on it then.

PS. And wouldn't you know it, I just ran across Google's Protocol Buffers, yet another solution for basically the same problem. I have my work cut out for me!

Enter JSON doesn't (excluding the hack of using Unicode newlines to separate your lines—DO NOT WANT!).

Anyway, I decided on writing a proof of concept for using YAML, and started it this evening. I haven't gotten terribly far, but far enough to run in to some issues reading files with yaml-cpp. Even reading the YAML 1.2 spec sample files. I'm not sure what's wrong. In any case, I hope to resolve this soon and that YAML works out. I'd really like to use it for the MUD!

I can parse the "complete example" code, though. Right down to each individual scalar. I used recursion to read the nodes until I hit a scalar value instead of their fancy operator>> overloading.