Category: Programming

This is more a case of posting here to remind myself, but maybe it will help someone out. I just ran into a case where QuickFIX/J seemed to be stacking up endless messages in a LinkedHashMap object (messageQueue) in SessionState.java (discovered using a heap dump and the helpful Memory Analysis Tool from Eclipse) and spent a few hours banging my head on the wall trying to figure out why. After I bothered to read the QFJ code, I understood why, but it’s non-obvious.

Essentially, QFJ’s Session class parses each incoming message, and then attempts to parse any “queued” messages from the aforementioned LinkedHashMap before doing it all again. The contents of the LinkedHashMap (in my case, built up to 500+ megabytes before bringing the JVM to flaming ruin) are only messages that result from a “sequence number too high” error. So if your QFJ application throws, for example, a repeated NPE while handling FIX messages, in certain cases this can trigger a “sequence number too high” problem and will result in FIX messages stacking up in that QFJ-side queue.

I’m not sure if this is really a bug in QFJ or not — I should have checked the QFJ event log sooner, and I might have solved this without noticing the pileup. I suspect that’s why there isn’t a lot of information to be easily found online about the issue – other people just debug their applications correctly.;) That said, it’s non-obvious to me that anything I do in my QFJ application should result in some map inside QFJ’s core classes growing out of control. Hence, noted for myself and anyone else who may care/be curious.:)

I’ve had occasion to play around with FreeSwitch lately and was looking at implementing a directory of extensions on the main menu, and so I set up Brian Snipes’ example Lua directory from the FreeSwitch wiki. I didn’t like the fact that the extension list was hardcoded, so I replaced the names{} table with the following bit of code. It’s ugly, but it’s my first 2-3 hours of messing with Lua. Anyway, it basically makes a system call to “grep” to pull the extensions out of your FS directory XML files using the effective_caller_id_* values which you may or may not have in each listing in the directory. Maybe this will help someone avoid having to reinvent the wheel completely.

Note: this presumes that each entry in the directory lives in its own users_[firstinitial][lastname].xml file. For example, I have users_nsimpson.xml containing the following two lines:

<variable name="effective_caller_id_name" value="Nate Simpson"/>
<variable name="effective_caller_id_number" value="1000"/>

You can hack the following code to match the particulars of your own installation. Also, I think string.gfind() has become string.gmatch() in Lua 5.1, so you may need to tweak that as well.


-- NS: build the names table from grep
directory_path = "/usr/local/freeswitch/conf/directory/default/"

namesearch = io.popen("grep effective_caller_id_ "..directory_path.."user_*.xml")
namelist = namesearch:read("*a")
namesearch:close()

names = {}

for thisuser, thisname in string.gfind(namelist, "user_(%a+)\.xml:%s+

I’ve recently been playing around with the SIMILE Timeline visualizer for XML data, which is pretty neat. I had one minor little gripe with it, which is that I wanted to be able to use data from an XML string in certain cases where I already was working in the context of an XSL stylesheet instead of hitting the server again just to present the same data in a different XML structure for the timeline. So I downloaded the code, told it NOT to use the bundled code in timeline-bundle.js by editing appropriately in timeline-api.js, and started experimenting with the sources.

What seems to have worked for me is the following modification [I added it just before the line starting: Timeline.DefaultEventSource.prototype.loadXML = function…] to sources.js (pardon the messy formatting here – you can correct the line breaks in the text editor of your choice 🙂 ):

Timeline.DefaultEventSource.prototype.loadXMLText = function(sometext) {
return Try.these(
// For Firefox, etc.
function() { return new DOMParser().parseFromString(sometext, ‘text/xml’); },
// For IE
function() { var xmldom = new ActiveXObject(‘Microsoft.XMLDOM’); xmldom.loadXML(sometext); return xmldom; },
// As a last resort, try loading the document from a data: URL. This is supposed to work in Safari. Thanks to Manos Batsis and his Sarissa library (sarissa.sourceforge.net) for this technique. (NS: – got this from timeline-helper.js in the GWT Timeline build)
function() { var url = “data:text/xml;charset=utf-8,” + encodeURIComponent(sometext); var request = new XMLHttpRequest(); request.open(“GET”, url, false); request.send(null); return request.responseXML; }
);
};

and in your calling script, replacing the line:

Timeline.loadXML(“example1.xml”, function(xml, url) { eventSource.loadXML(xml, url); });

with the following:

eventSource.loadXML(eventSource.loadXMLText(xmlstring,), “http://path.to.script/goes/here”);

Replace “xmlstring” with either a quoted string of XML or a string variable, and edit your path accordingly. Hope it works for you – it seems to be working fine for me!

If you are running Linux and you change from a DHCP setup to a static IP address on your local network, and then at some future time your ISP changes your external IP address and your internet slows to a crawl, you might find it useful to know that your resolv.conf file probably contains the old DNS entries set up back when you were using DHCP. It’s a simple matter to change it, but knowing the problem is 99% of this battle. Maybe this will help someone else out there some day who wonders why a Windows box on the same network is chugging along happily but your Linux box is choking. Happy surfing!

I was pretty happy to discover that the latest releases of MySQL (5.0+) allow for creating VIEWs and offer improved subqueries. I’ve been trying to incorporate some of this into my latest little game project, but a few irritations in MySQL are making it a miserable process.

The biggest problem is that the new MySQL VIEWs automatically resort to TEMPTABLE if you throw a GROUP BY into a query. As a result, it’s impossible to create certain types of VIEWs on very large tables, because your logical WHERE restriction on the VIEW behaves as if you issued GROUP BY…HAVING on the underlying query instead of WHERE somerestriction[s]…GROUP BY. If your query would return 10,000 rows (yes, I know that’s only relatively “very large” but I need to keep my query result times in the sub-second range and I only need 32 of those rows) before HAVING is applied, well, that’s what you get with the VIEW. For example:

CREATE VIEW myview AS SELECT foo.ID FROM foo_table foo GROUP BY foo.bar;

In that little snippet, you’d think that you could “SELECT * FROM myview WHERE foo.ID = 1” and restrict the result set, but the WHERE restriction is applied as a HAVING. This is really, really unfortunate, as it makes the VIEW painfully, unusably slow when you have more than a tiny handful of rows.

The second annoyance is that while MySQL supports multi-table UPDATEs, it behaves in an illogical (to me) fashion when trying to do the following:

UPDATE foo JOIN bar ON somecondition SET foo.somefield = foo.somefield + 1;

The logical behavior seems to me to be that for each “bar”, “somefield” would get incremented by 1. However, in practice, MySQL updates “somefield” once and ignores the rest of the results returned by the “bar” JOIN. It’s almost the same problem discussed here and here, but dumping/reloading the tables seems irrelevant (as does my MySQL version – 5.0.22 currently) and I seem to be bumping up against a restriction briefly mentioned at the bottom of the MySQL “UPDATE Syntax” documentation page – “Currently, you cannot update a table and select from the same table in a subquery.” [Emphasis mine] I’m not the only person to have raised questions about UPDATE’s odd multi-table behavior.

I was hoping to use something like the second to mitigate the problems created by the VIEW shortcomings, but that doesn’t seem to be happening. The best solution I’ve encountered seems to be to use group functions in a subquery as per:

UPDATE foo
(SELECT SUM(somevalue), id FROM bar WHERE somecondition GROUP BY something) bar_alias
SET foo.somefield = foo.somefield + bar_alias.somevalue
WHERE foo.id = bar_alias.id;


That’s much less elegant and less flexible than using either an UPDATE…JOIN myview or UPDATE…join_query which I’d hoped to be able to do.

Here’s to hoping that the VIEW gets optimized better for the future. It would definitely make my life easier to be able to use GROUP BY in a VIEW without choking up my system. Or maybe “Currently” will become “Long ago” and I can use self-joining table UPDATEs, which is better than nothing.

And if you happen to know any way around either of these that doesn’t involve eliminating the GROUP BY on a VIEW or using a SUM() on the UPDATE, please tell me!