Davor Josipovic Just another WordPress blog – rather tryout

27/04/2013

Circumvent the 260 MAX_LENGTH path in PowerShell with junctions

Filed under: PowerShell,Programming — Tags: , — Davor @ 18:21

This is a function I wrote to circumvent the 260 char path MAX_LENGTH in Win32 API on which all of the cmdlets in PowerShell are based.

Usage: $short_path = Shorten-Path “some-long-path” “temporary-directory-path”

What the function will do is simply make a junction point in the temporary directory, and return a shorter path that points to the same place as the original long path, but through the junction point. This is only if the path is too long: if not, then it will return the original path.

# Returns a shortened path made with junctions to circumvent 260 path length in win32 API and so PowerShell
function Shorten-Path {
	[CmdletBinding()]
	param( 
		[Parameter(Mandatory=$true,
				   Position=0,
				   ValueFromPipeline=$true,
				   HelpMessage="Path to shorten.")]
		[string]$Path,
		[Parameter(Mandatory=$true,
				   Position=1,
				   ValueFromPipeline=$false,
				   HelpMessage="Path to existing temp directory.")]
		[string][ValidateScript({Test-Path -LiteralPath $_ -PathType Container})]$TempPath    
	)
 
	begin {
		# Requirements check
		if (-not $script:junction) {$script:junction = @{};}
		$max_length = 248; # this is directory max length; for files it is 260.
	}
 
	process {
		# First check whether the path must be shortened.
		# Write-Warning "$($path.length): $path"
		if ($Path.length -lt $max_length) {
			Write-Debug "Path length: $($Path.length) chars."; 
			return $Path;
		}
 
		# Check if there is allready a suitable symlink	
		$path_sub = $junction.keys | foreach { if ($Path -Like "$_*") {$_} } | Sort-Object -Descending -Property length | Select-Object -First 1;
		if ($path_sub) {
			$path_proposed = $Path -Replace [Regex]::Escape($path_sub), $junction[$path_sub];
			if ($path_proposed.length -lt $max_length) {
				# assert { Test-Path $junction[$path_sub] } "Assertion failed in junction path check $($junction[$path_sub]) for path $path_sub.";
				return $path_proposed;
			}
		}
 
		# No suitable symlink so make new one and update junction
		$path_symlink_length = ($TempPath + '\' + "xxxxxxxx").length;
		$path_sub = ""; # Because it is allready used in the upper half, and if it is not empty, we get nice errors...
		$path_relative = $Path;
		# Explanation: the whole directory ($Path) is taken, and with each iteration, a directory node is taken from
		# $path_relative and put in $path_sub. This is done until there is nothing left in $path_relative.
		while ($path_relative -Match '([\\]{0,2}[^\\]{1,})(\\.{1,})') {
			$path_sub += $matches[1];
			$path_relative = $matches[2];
			if ( ($path_symlink_length + $path_relative.length) -lt $max_length ) {
				$tmp_junction_name = $TempPath + '\' + [Convert]::ToString($path_sub.gethashcode(), 16);
				# $path_sub might be very large. We can not link to a too long path. So we also need to shorten it (i.e. recurse).
				$mklink_output = cmd /c mklink /D """$tmp_junction_name""" """$(Shorten-Path $path_sub $TempPath)""" 2>&1;
				$junction[$path_sub] = $tmp_junction_name;
				# assert { $LASTEXITCODE -eq 0 } "Making link $($junction[$path_sub]) for long path $path_sub failed with ERROR: $mklink_output.";
				return $junction[$path_sub] + $path_relative;
			}
		}
 
		# Path can not be shortened...
		# assert $False "Path $path_relative could not be shortened. Check code!"
	} 
 
	end {}
}

In $junction variable, all the junction points are stored, so you can remove them afterwards with:

foreach ($link in $junction.values) {
	$rmdir_error = cmd /c rmdir /q """$link""" 2>&1;
	if ( $LASTEXITCODE -ne 0 ) { Write-Warning "Removing link $link failed with ERROR: $rmdir_error." };
}
$junction.clear();

I am also using the assert function which you can find here.

24/11/2010

System-wide events and Qt

Filed under: C++,Programming,Qt — Tags: , , — Davor @ 23:02

Few days back I wanted to create a macro recorder for the desktop. I though I could do this from within the Qt framework which has a neat high-level platform-independent event system that I thought I could incorporate with widgets to make a nice user interface. Well, that’s what I though at least… Truth is, Qt doesn’t offer any means for catching system-wide events. Eventually, I managed to put together a little command line application that can record and replay/loop the mouse and keyboard input messages based on the Windows API. What you will read next is my explanation why catching system-wide events is impossible from within the Qt framework, what the Windows API offers for this purpose, and a proposal on how system-wide events could be incorporated in the Qt event system.

Background

Windows and Qt applications are event driven, which means that they wait for the system to pass events to them upon which they can act. Simply put, those events are passed as “messages” to application windows through a callback function named the “window procedure”.

For example, when a user moves a mouse, the mouse device driver places the recorded movement in the system’s message queue. The system then determines the destination window and “posts” this message to the correct “thread message queue” (i.e. the message queue of the thread which created the destination window). Our Qt application has only access to these messages from its thread message queue. These messages are called “spontaneous events” in the Qt documentation. (Note: there exist also “sent” messages which bypass the message queue, but which we do not consider here.)

Now as one can see, the system is very selective on which mouse and keyboard events our Qt application will get. For example, our window will not get any keyboard messages if it is not in focus, nor will it get any mouse events if the mouse is not inside the window borders. So how can we access those events from the Qt framework? To my knowledge Qt doesn’t offer any ready made functions for this, so we will have to find some other way: directly accessing the Windows API.

Hooks

The Windows API offers a mechanism called a “hook” which we can hook somewhere in the message-handling mechanism and which will intercept and reroute the messages to some “hook procedure” we define. We define a hook with the API SetWindowsHookEx procedure. This procedure allows us to specify the “hook type” which defines the kind of messages to be intercepted (like WH_MOUSE_LL, WH_KEYBOARD_LL, WH_SHELL), the “hook scope” which can be thread or global (i.e. system), and the hook procedure with the following predefined signature:

 LRESULT CALLBACK HookProc(int nCode, WPARAM wParam, LPARAM lParam);

So to get you started here is a very basic application which uses the above described mechanism an which can be used as a starting point for building applications which are able to monitor system events.

#include <QtGui/QApplication>
#include <QDebug>
 
#include <windows.h>
 
LRESULT CALLBACK HookProc(int nCode, WPARAM wParam, LPARAM lParam)
{
    // process event...
    qDebug() << nCode << wParam << lParam;
 
    return CallNextHookEx(NULL, nCode, wParam, lParam);
}
 
int main(int argc, char *argv[])
{
    QApplication a(argc, argv);
 
    if (SetWindowsHookEx(WH_KEYBOARD_LL, HookProc, qWinAppInst(), NULL) == 0)
        qDebug() << "Hook failed for application instance" << qWinAppInst() << "with error:" << GetLastError();
 
    return a.exec();
}

For more information, the following links might be interesting:

I’ve put together a little command line utility that is able to record and replay/loop the recording based on the above explanation. Sourcecode is available here. To record: “winrec -record /f “recording.txt””. To play: “winrec -play /f “recording.txt” /repeat 0″. To stop any of these, just press Crtl+Esc.

I think integration with an Qt would probably be straightforward when used with QAbstractEventDispatcher, although I didn’t have time to test it.

11/08/2010

“Undefined reference to vtable” for QObject-derived classes in the implementation (.cpp) file.

Filed under: Programming,Qt — Tags: , — Davor @ 18:14

“undefined reference to vtable” seems to be a common problem with Qt’s meta-object system in combination with GCC.

The main cause due to GCC is described here: http://gcc.gnu.org/faq.html#vtables

But with QObject it gets a bit more complex. When you use the macro Q_OBJECT, you define some virtual methods which might trigger the above GCC error under certain circumstances. So make sure your use of the Q_OBJECT macro fulfills the following requirements:

  1. Make sure the Q_OBJECT macro is present in the definition of all QObject-derived classes
  2. Make sure you define your QObject-derived classes in your header files ONLY
  3. Make sure all of your header files are listed in your .pro file in the HEADERS-list
  4. Run qmake every time you add Q_OBJECT to one of your classes or modify your .pro file

(Source: http://www.theirishpenguin.com/2007/07/01/qobject-qmake-and-sadness-undefined-reference-to-vtable/)

Not fulfilling point 2:
You get the “undefined reference to vtable” error if you put your QObject-derived class in the implementation file because moc will not “add” the implementation of the virtual functions to the cpp file. Apparently you can enforce this by adding “#include cpp_file_name.moc” to your “cpp_file_name.cpp” file. Moc will detect this and will generate a “cpp_file_name.moc” file.

29/06/2010

PostgreSQL table audit in plpgsql

Filed under: Postgres,Programming — Tags: , — Davor @ 15:27

I recently had to implement a logging system for our database. Googling resulted in a few interesting ways to do this. The best I have found so far is from Lorenzo Alberton. I recommend you glance through that, and also an extension/improvement of it by James Gardner.

Now, Lorenzo’s implementation has a few problems. Most notably: not all of us want to install Tcl. So, I have “rewritten” it in plpgsql. This is possible since in the meantime the EXECUTE command was extended with the USING clause.

Some extra features are added also:

  • Support for multi-field PK’s.
  • Logging of PK’s.
  • Changes are detected by default type operators instead of textbased operators.

Note: I am using a slightly different naming convention, so you should adjust the names before using it with extra functions from James.

Note2: This is still a (working) tryout. Comments on improvement are welcome.

Update: fnc_audit(…) is used by trg_audit()

CREATE OR REPLACE FUNCTION private.tfc_audit()
  RETURNS TRIGGER AS
$BODY$ 
 
/*!
 Dynamic query exectution through plpgsql EXECUTE command makes this function possible.
 Some complications which can araise using this command are described here:
  - http://archives.postgresql.org/pgsql-general/2008-05/msg00314.php
*/ 
 
DECLARE 
 
_PK_NAMES CONSTANT VARCHAR[] := ARRAY(SELECT a.attname AS pk_name FROM pg_class c, pg_attribute a, pg_index i
     WHERE c.oid = i.indrelid
     AND a.attrelid = i.indexrelid
     AND a.attisdropped = FALSE /* such column names are like "........pg.dropped.1........" and are invalid */
     AND a.attnum > 0 /* i.e is not a system column like OID: they have (arbitrary) negative numbers */ 
     AND i.indisprimary = TRUE /* is pk */
     AND c.oid = TG_RELID /* regclass takes current schema into consideration! */); 
 
_FIELD_NAMES CONSTANT VARCHAR[] := ARRAY(SELECT a.attname AS pk_name FROM pg_class c, pg_attribute a
     WHERE c.oid = a.attrelid
     AND a.attisdropped = FALSE /* such column names are like "........pg.dropped.1........" and are invalid */
     AND a.attnum > 0 /* i.e is not a system column like OID: they have (arbitrary) negative numbers */ 
     AND c.oid = TG_RELID ); /* regclass takes current schema into consideration! */ 
 
_PK_VALUES VARCHAR[]; 
 
BEGIN 
 
-- this function works only when marked as AFTER trigger!
PERFORM assert(TG_WHEN = 'AFTER', 'Wrong execution of fnc_audit(): should only be executed AFTER an event!');
-- this function works only when marked as FOR EACH ROW trigger!
PERFORM assert(TG_LEVEL = 'ROW', 'Wrong execution of fnc_audit(): should only be executed FOR EACH ROW!'); 
 
-- table sanity check
PERFORM assert(TG_TABLE_NAME IS NOT NULL, 'TG_TABLE_NAME IS NULL');
PERFORM assert(_PK_NAMES IS NOT NULL AND array_length(_PK_NAMES, 1) IS NOT NULL, 'Table ' || TG_TABLE_NAME || ' can not be logged: it has no PK!');  -- FOR-loop will otherwise throw "upper bound of FOR loop cannot be null" 
 
-- skip changes on audit_table: otherwise endless loop
IF TG_TABLE_NAME = 'tbl_audits' THEN
 RETURN NULL; -- result is ignored since this is an AFTER trigger
END IF; 
 
-- find PK value
FOR i IN 1 .. array_length(_PK_NAMES, 1)
LOOP
 DECLARE
  tmp text;
  tmp_record record;
 BEGIN
  CASE TG_OP
  WHEN 'UPDATE', 'INSERT' THEN tmp_record = NEW;
  WHEN 'DELETE' THEN tmp_record = OLD;
  END CASE; 
 
  EXECUTE 'SELECT $1.' || quote_ident(_PK_NAMES[i]) INTO STRICT tmp USING tmp_record;
  _PK_VALUES[i] := tmp;
 END;
END LOOP; 
 
-- PK values sanity check
PERFORM assert(_PK_VALUES IS NOT NULL AND array_length(_PK_VALUES, 1) IS NOT NULL, '_PK_VALUES IS NULL'); -- FOR-loop will otherwise throw "upper bound of FOR loop cannot be null"
PERFORM assert(array_length(_PK_VALUES, 1) = array_length(_PK_NAMES, 1), '_PK_VALUES count not equal to _PK_NAMES count in table ' || TG_TABLE_NAME); 
 
-- find fields to be logged
FOR i IN 1 .. array_length(_FIELD_NAMES, 1)
LOOP
 -- field_name sanity check
 PERFORM assert(_FIELD_NAMES[i] IS NOT NULL, '_FIELD_NAMES[' || i || '] IS NULL'); 
 
 DECLARE
  _field_name CONSTANT VARCHAR := quote_ident(_FIELD_NAMES[i]);
  _distinct BOOLEAN;
  _old text;
  _new text;
 BEGIN 
 
 CASE TG_OP --fall through is unfortunately not supported
  WHEN 'UPDATE' THEN
   -- compare the two fields according to the type-default functions.
   -- note that <> returns false on NULL input and IS DISTINCT FROM does not, but only if both inputs are NULL!
   EXECUTE 'SELECT $1.' || _field_name || ' IS DISTINCT FROM $2.' || _field_name || ';' INTO STRICT _distinct USING OLD, NEW;
   CONTINUE WHEN NOT _distinct; 
 
   EXECUTE 'SELECT $1.' || _field_name || ';' INTO STRICT _new USING NEW;
   EXECUTE 'SELECT $1.' || _field_name || ';' INTO STRICT _old USING OLD;
  WHEN 'INSERT' THEN
   EXECUTE 'SELECT $1.' || _field_name || ';' INTO STRICT _new USING NEW;
  WHEN 'DELETE' THEN
   EXECUTE 'SELECT $1.' || _field_name || ';' INTO STRICT _old USING OLD;
  ELSE
   RAISE EXCEPTION 'Unhandled TG_OP in fnc_audit(): %', TG_OP;
 END CASE; 
 
 CONTINUE WHEN _new IS NULL AND _old IS NULL; 
 
 PERFORM private.fnc_audit(statement_timestamp()::TIMESTAMP WITHOUT TIME zone, SESSION_USER::VARCHAR, TG_TABLE_NAME::VARCHAR, _field_name, _PK_NAMES, _PK_VALUES, TG_OP::private.table_audit_mod_type, _old, _new);
 
 END;
END LOOP; 
 
RETURN NULL; -- result is ignored since this is an AFTER trigger
END; 
 
$BODY$
  LANGUAGE 'plpgsql' VOLATILE SECURITY DEFINER;

A specific function for inserting into tbl_audits is used:

CREATE OR REPLACE FUNCTION private.fnc_audit(_audi_timestamp TIMESTAMP WITHOUT TIME zone, _audi_user name, _audi_table name, _audi_field name, _audi_pk_name CHARACTER VARYING[], _audi_pk_value CHARACTER VARYING[], _audi_mod_type private.table_audit_mod_type, _audi_value_old text, _audi_value_new text)
  RETURNS void AS
$BODY$
DECLARE
BEGIN 
 
INSERT INTO private.tbl_audits(audi_timestamp, audi_user, audi_table, audi_field, audi_pk_name, audi_pk_value, audi_mod_type, audi_value_old, audi_value_new)
 VALUES (_audi_timestamp, _audi_user, _audi_table, _audi_field, _audi_pk_name, _audi_pk_value, _audi_mod_type, _audi_value_old, _audi_value_new); 
 
END;
$BODY$
  LANGUAGE 'plpgsql' VOLATILE;

TG_OP is stored as an ENUM (= more efficient) so we need a new type:

CREATE TYPE private.table_audit_mod_type AS ENUM ('INSERT', 'UPDATE', 'DELETE', 'TRUNCATE');

Here is a very important helper function:

CREATE OR REPLACE FUNCTION assert(BOOLEAN, CHARACTER VARYING)
  RETURNS void AS
$BODY$
BEGIN
 IF NOT $1 OR $1 IS NULL THEN
   IF $2 IS NOT NULL THEN 
     RAISE EXCEPTION 'Assert failure: %', $2;
   END IF;
   RAISE NOTICE 'Assert. Message is null';
 END IF; 
END;$BODY$
  LANGUAGE 'plpgsql' VOLATILE;

And finaly the audit table:

CREATE TABLE private.tbl_audits
(
  audi_timestamp TIMESTAMP WITHOUT TIME zone NOT NULL,
  audi_user CHARACTER VARYING(30) NOT NULL,
  audi_table CHARACTER VARYING(45) NOT NULL,
  audi_field CHARACTER VARYING(30) NOT NULL,
  audi_pk_name CHARACTER VARYING(30)[] NOT NULL,
  audi_pk_value CHARACTER VARYING(60)[] NOT NULL,
  audi_mod_type private.table_audit_mod_type NOT NULL,
  audi_value_old text,
  audi_value_new text,
  CONSTRAINT tbl_audits_check_pk CHECK (array_length(audi_pk_name, 1) = array_length(audi_pk_value, 1))
);

A more storage-efficient way to store logged data would be to make an audi_changeset column which consists of an array of changed fieldnames, old values and new values for each PK during a statement execution. It would be equivalent to the following view:

CREATE OR REPLACE VIEW vew_audit_changesets AS
 SELECT tbl_audits.audi_timestamp, tbl_audits.audi_user, tbl_audits.audi_table, tbl_audits.audi_pk_name, tbl_audits.audi_pk_value, tbl_audits.audi_mod_type, ARRAY[array_agg(tbl_audits.audi_field), array_agg(tbl_audits.audi_value_old)::CHARACTER VARYING[], array_agg(tbl_audits.audi_value_new)::CHARACTER VARYING[]] AS audi_changeset
   FROM tbl_audits
  GROUP BY tbl_audits.audi_timestamp, tbl_audits.audi_user, tbl_audits.audi_table, tbl_audits.audi_pk_name, tbl_audits.audi_pk_value, tbl_audits.audi_mod_type;

fnc_audit() is easily adjustable to store logs to such a table. But I wonder whether this is worth the fuss? Most of the queries run on vew_audit_changesets would first require the “unnesting of array audi_changeset to sets” (i.e. a structure which resembles tbl_audits). Why? Because SQL is set-oriented, not array-oriented. In other words, it seems easy to convert from tbl_audits structure to vew_audit_changesets structure, but how does one go from vew_audit_changesets structure to tbl_audits structure, and is this conversion efficient? Is it worth to save logs in a vew_audit_changesets-like table?

Powered by WordPress