Davor Josipovic Just another WordPress blog – rather tryout

02/02/2017

Primal Active-Set method for constrained convex quadratic problem

Filed under: Algorithms — Tags: , — Davor @ 10:05

I made an implementation of the Active-Set Method for Convex QP as described in Nocedal, J. e.a., Numerical Optimization, 2ed, 2006, p.472. The code in R can be found on Github. Output with two examples from the book can be found here.

There is an other free package named quadprog that does the same but with the limitation of only accepting positive definite matrices. This can be tweaked to work with positive semi-definite matrices. For example one can convert a positive semi-definite matrix to its nearest definite one with Matrix::nearPD() function. To cope with semi-definiteness in the Active-Set method, Moore-Penrose generalized inverse is used for solving the KKT problem.

Note that the Active-Set method must start with an initial feasible point. Understanding the problem is usually enough to calculate one. Nocedal describes a generic “Phase I” approach for finding a feasible point on p.473.

28/11/2016

What is so semantic in Semantic web anyway?

Filed under: Philosophy — Tags: , — Davor @ 11:09

Semantic web. Semantic? This eluded me back from the time I first heard the term. It is concerned with the meaning and not as much with the structure of data. But how?

The term “semantic” was coined by Tim Berners-Lee for a web of data that can be processed by machines. But that doesn’t tell us why it is called semantic. What has meaning to do with machines and processing?

After I saw this MIT presentation on the suject it dawned upon me that there is an interesting equivalence between how truth is defined in Semantic web, and the way Donald Davidson defined meaning in his “theory of meaning”. His ideas go back to the 60s, and are based on Tarski’s theory of truth from the 30s. So here is my colloquized way of explaining the intuition of why the Semantic web is actually semantic.

The semantinc web in the MIT presentation is defined as:

XML + RDF + Ontologies + Inference rules = Semantic web!

You wonder why all this equates to “Semantic”?

Suppose you have a set of entities, lets say {Socrates, man, mortal}. Suppose don’t know about nothing else than those three words. Suppose that that is your Ontology. That is your world. Note that having an Ontology is a constitutive requirement for a Semantic web. So are the Inference rules. Suppose we have only one Inference rule: if A is B, and B implies C then A is C. The third constitutive component (RDF) consists of  some true statements about your set of objects which are contained in the subjectpredicateobject structure. For example: Socrates (subject) is (predicate) a man (object) and men (subject) are (predicate) mortal (object). XML is used to describe all of the other three so a machine can read and interpret them. Now given the preceding example, what the machine can do is deduce the truth of a statement like: Socrates is a man, all men are mortal, therefore Socrates is mortal (implication from Inference rule). Now remember (!), given its Ontology, Inference rules and RDF, the machine knows which statement is true, and which is not true. The machine knows that the statement Socrates is mortal is true!

Now, where is semantics in all this? Well, Davidson proposed the idea that truth and meaning are equivalent. If you have one, then you have the other too. Note that given the RDFs, Ontology and Inference rules, the machine actually knows when any statement is true. Thus the machine knows the meaning of the statement.

That too quick? Not convinced? Well, suppose I ask you whether the following statement is true: “Socrates je covjek”. Can you assess the truth? Not if you don’t know Croatian – which I assume you do not for the sake of the example. But suppose I tell you that the statement “Socrates je covjek” is true under all conditions under which “Socrates is a man” is true (i.e. the two statements are equivalent). Since you know that “Socrates is a man” is true (i.e. it is explicitly stated in your RDF), and that “Socrates je covjek” is true whenever “Socrates is a man” is true, then you can perfectly say that you understand “Socrates je covjek” and thus know its meaning. In other words, if you know the truth conditions of “Socrates je covjek”, you understand it. The same can be done with “Socrates is mortal”. Although it is not explicitly stated in the RDF, the machine can deduce its truth. So if “Chapa muju koki” is equivalent to “Socrates in mortal”, then you can say that you understand it. That is the mechanism behind the Semantic web. Knowing the truth of a statement implies knowing the meaning of the statement – and vice versa.

Thus the web is semantic.

Now you can ask whether meaning is not more than only truth. That is a difficult question for which I can not go into much details here. My philosophical days are unfortunately numbered. But a good starting point on this subject is probably here. My own intuition is that meaning as defined above is only a subset of what we understand under meaning. Thus, the web is semantic, but only up to a certain degree…

30/08/2014

Transform any (binary) function to an aggregate in pure SQL

Filed under: Oracle — Tags: , — Davor @ 10:44

Few days back I needed an aggregate counterpart of a BITOR function. Unfortunately the Oracle database doesn’t have one.

So how do I make one? There are three options here:

  1. Write your own aggregate function. Here is one way to do it.
  2. Rewrite it in function of an other aggregate function. For example PRODUCT() can be rewritten as EXP(SUM(LN())) (cf. infra). But there is no obvious way for writing an aggregate BITOR() in function of existing Oracle functions.
  3. Simulate aggregation in pure SQL. If you for example have 4 elements {a, b, c, d}, you know that their BITOR aggregate is: BITOR(a,BITOR(b,BITOR(c,d))). Since SQL:1999 we have recursion in SQL. So why write an aggregate function if you can compute it within SQL?

This blogpost is all about the third option. I wanted to see (I) whether it is possible and (II) whether I can generalize it in a concise manner for all binary functions. I also couldn’t find any information about this on the Internet, so that is why I am writing this. It turns out (I) is true, and (II) also, albeit with complications. What I didn’t expect is very bad performance. So the solution below is only for educational use.

Aggregation with + operator

Let’s start with summation. Summation (+) is a binary operator/function that is easy to understand and easy to verify with the aggregate SUM() function.

Suppose this is your table:

CREATE TABLE t4 (
  gr NUMBER(10), -- group
  nr NUMBER(10)  -- number
);

Insert some values in it

INSERT INTO t4 VALUES (1,10);
INSERT INTO t4 VALUES (1,20);
INSERT INTO t4 VALUES (1,30);
INSERT INTO t4 VALUES (1,40);
INSERT INTO t4 VALUES (1,50);
INSERT INTO t4 VALUES (2,20);
INSERT INTO t4 VALUES (2,30);

What we have is this:

        GR         NR
---------- ----------
         1         10 
         1         20 
         1         30 
         1         40 
         1         50 
         2         20 
         2         30 

The way to sum-aggregate these numbers is by adding a new column which will compute the sum of the current number and the previous sum:

        GR         NR RECURSIVE_SUM
---------- ---------- -------------
         1         10            10 
         1         20            30 
         1         30            60 
         1         40           100 
         1         50           150 
         2         20            20 
         2         30            50 

Finally I just have to take the last step in each group: 150 for gr = 1 and 50 for gr = 2. The recursive query that does all the above and uses only the (+) operator is the following:

WITH sel AS (
  SELECT t4.nr, t4.gr, ROW_NUMBER() OVER (partition BY gr ORDER BY nr) AS rn
  FROM t4
), rec(gr, r_out, rn, nr) AS (
    SELECT gr, nr, rn, nr FROM sel WHERE rn = 1
    UNION ALL
    SELECT sel.gr, rec.r_out + sel.nr, rec.rn + 1, sel.nr FROM rec, sel WHERE sel.rn = rec.rn + 1 AND sel.gr = rec.gr
), gr AS (
  SELECT gr, COUNT(gr) AS MAX FROM sel GROUP BY gr 
)
SELECT gr.gr, r_out SUM FROM gr INNER JOIN rec ON (gr.max = rec.rn AND gr.gr = rec.gr) ORDER BY gr.gr;
        GR        SUM
---------- ----------
         1        150 
         2         50 

This is equivalent to:

SELECT SUM(nr) SUM FROM t4 GROUP BY gr ORDER BY gr;
        GR        SUM
---------- ----------
         1        150 
         2         50 

Aggregation with * operator

Now let’s try multiplication. Just like summation, a product (*) is a binary operator/function. Now, to simulate the PRODUCT() function with the binary * function, you only have to change the r_out field in the recursive query:

WITH sel AS (
  SELECT t4.nr, t4.gr, ROW_NUMBER() OVER (partition BY gr ORDER BY nr) AS rn
  FROM t4
), rec(gr, r_out, rn, nr) AS (
    SELECT gr, nr, rn, nr FROM sel WHERE rn = 1
    UNION ALL
    SELECT sel.gr, rec.r_out * sel.nr, rec.rn + 1, sel.nr FROM rec, sel WHERE sel.rn = rec.rn + 1 AND sel.gr = rec.gr
), gr AS (
  SELECT gr, COUNT(gr) AS MAX FROM sel GROUP BY gr 
)
SELECT gr.gr, r_out product FROM gr INNER JOIN rec ON (gr.max = rec.rn AND gr.gr = rec.gr) ORDER BY gr.gr;
        GR    PRODUCT
---------- ----------
         1   12000000 
         2        600 

We can verify the above outcome with the aggregate PRODUCT() function rewritten as EXP(SUM(LN())). With a little algebra you can figure out why the equation holds. Here is the statement:

SELECT gr, EXP(SUM(LN(nr))) AS product FROM t4 GROUP BY gr ORDER BY gr;
        GR    PRODUCT
---------- ----------
         1   12000000 
         2        600 

Aggregation with BITOR function

By now it should be clear how to adjust the recursive query to simulate aggregation for any binary function: We only have to adjust the r_out field. Now let’s try to simulate the aggregated BITOR function. Because there is no BITOR we can rewrite it as BITOR(x,y) = (x+y)-BITAND(x,y);

WITH sel AS (
  SELECT t4.nr, t4.gr, ROW_NUMBER() OVER (partition BY gr ORDER BY nr) AS rn
  FROM t4
), rec(gr, r_out, rn, nr) AS (
    SELECT gr, nr, rn, nr FROM sel WHERE rn = 1
    UNION ALL
    SELECT sel.gr, (rec.r_out + sel.nr) - BITAND(rec.r_out, sel.nr), rec.rn + 1, sel.nr FROM rec, sel WHERE sel.rn = rec.rn + 1 AND sel.gr = rec.gr
), gr AS (
  SELECT gr, COUNT(gr) AS MAX FROM sel GROUP BY gr 
)
SELECT gr.gr, r_out BITOR FROM gr INNER JOIN rec ON (gr.max = rec.rn AND gr.gr = rec.gr) ORDER BY gr.gr;
        GR      BITOR
---------- ----------
         1         62 
         2         30 

Note 1: all the above recursive code is very inefficient. Tables exceeding 100 values will have a large performance impact. The problem is the statement following the UNION ALL: it has to select the right value(s) for each recursive iteration step. When I find more time I’ll try to optimize it. In the meantime, here is a query to fill the table with some dummy values for testing:

INSERT INTO t4
SELECT round(dbms_random.value(1,5)), round(dbms_random.value(1,99)) FROM DUAL
CONNECT BY level <= 100;

16/08/2013

Versioned hardlinked shadowed backup solution with PowerShell

Filed under: PowerShell,Programming — Tags: , — Davor @ 15:36

I don’t like the fact that there are so many backup programs, most of which are expensive closed source solutions. All have their own (mostly) undocumented and proprietary archive systems, that are hardly usable without the software that made them. All this… while Windows 7 offers enough technology under the hood to make an open source scripted solution possible, where the files are versioned, incremental and hardlinked (thus saving space), while the versioned backup contents can be viewed in Explorer. No extra software is needed. The current version of the script you can find on GitHub.

Here is an overview:

Root of the backup folder.

Root of the backup folder. The contents are hardlinked, which means that 10 identical files backed up at different times take only the space of 1 file.

The includelist file for the backup might look like this:

D:\Hardware\*
M:\Music\Playlists\*
M:\Pictures\*
W:\Research\*
W:\Server\apache\*\*.conf

And this is an example of how to run the backup script:

.\ps-backup.ps1 -Backup -BackupRoot "W:\Backups\Archive" -SourcePath "W:\Scripts\ps-backup\include_list.txt" -ExclusionList "W:\Scripts\ps-backup\exclude_list.txt"

23/07/2013

How to make Mass Effect (Steam) start on Windows 7/8 without admin rights

Filed under: Gaming — Tags: , — Davor @ 21:32

First the reason why it will not start, and then the solution.
I traced the process during startup. It seems MassEffect.exe tries to write to HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\AGEIA Technologies, but it doesn’t have enough privileges. Running it as admin for the first time will set the appropriate values. On subsequent runs these values will only be read, so no admin rights are needed.

Solution
Either run Steam elevated (i.e. with admin rights) the first time you run Mass Effect, or make the HKLM key yourself and give the “User” group full rights on it. In the second case, after you run Mass Effect for the first time it will make “enableLocalPhysXCore” and “EpicLocalDLLHack” REG_BINARIES, and add appropriate values to them. Value for “enableLocalPhysXCore” is system dependent (i.e. not generic!)

Note: simply running MassEffect.exe with admin rights will not suffice. The reason is that once MassEffect.exe is executed with admin rights, it will run Steam for authentication and exit. Then Steam will run MassEffect.exe again, but it is not elevated this time, so it will not be able to write to HKLM. You either need to run Steam elevated, or set “run this program as administrator” in compatibility settings of MassEffect.exe.

02/06/2013

Ignoring device I/O errors during copy with PowerShell

Filed under: PowerShell,Programming — Tags: , , — Davor @ 19:14

I had a failing hard drive with lots of RAR files with recovery records. Problem was that I couldn’t get the files off the hard drive with tools like robocopy and xcopy because the drive had many bad sectors resulting in CRC and I/O device errors. I also couldn’t repair the drive with chkdsk /R because the bad sectors kept reappearing.

I also tried Unstoppable Copier, but on best settings it seems to use 0,1% of the file size per read operation, which results in large data corruption when that read operation fails even if there are only a few bad sectors.

So I wrote a little PowerShell script which will copy the source files and replace the unreadable data with zero’s, as accurately as the partition’s cluster size allows. It will also make a XML file per file where it will store bad sector data positions for further reference. I have placed the script code on GitHub.

Note on reading bad sector data

When a hard drives fails to read a sector due to a CRC error, it doesn’t give back any data. Instead it raises an error: I/O device error. With old hard drives it was possible to issue the READ LONG command to skip the error correction part, and simply give back the data. Some programs like Spinrite use this to recover data based on re-read statistics. (If on hundred consecutive re-read operations for sector X your starting bits are “1011…”, the it is reasonably safe to assume those bits are not corrupted.) But the READ LONG command doesn’t work with modern hard drives. Modern hard drives either succeed or fail a read operation. There is no third option. What I have noticed during my own recovery is that even though the hard drive fails reading a sector the first time, there is some chance it will succeed on a retry. I have seen it succeed even after 100 fails! That’s why the -MaxRetries is so important. But setting it too high will greatly slow don the recovery process in case a sector is truly unreadable.

Notes on script usage

  1. First you need to download the script. Here is a link to the file. Save it as Force-Copy.ps1 to, for example, your desktop. (Make sure the extension is correct: ps1!) (There is also a more simple script available at stackoverflow.)
  2. This is a PowerShell script, so it has to run under the PowerShell environment. So run PowerShell and navigate it to your desktop directory. For example, run cmd, type cd desktop and then type powershell -ExecutionPolicy bypass. (Execution policy needs to be changed because PowerShell will not allow scripts to be run by default. With bypass, the script is not blocked and there are no warnings or prompts. This is because the script is unsigned. Read more about PowerShell execution policy and signing here.)
  3. Now, to see some examples of use, type Get-Help .\Force-Copy.ps1 -Examples. The most simple command to copy a file from one to an other location would be:
    .\Force-Copy.ps1 -SourceFilePath "C:\bad_file.txt" -DestinationFilePath "W:\recovered_file.txt"

27/04/2013

Circumvent the 260 MAX_LENGTH path in PowerShell with junctions

Filed under: PowerShell,Programming — Tags: , — Davor @ 18:21

This is a function I wrote to circumvent the 260 char path MAX_LENGTH in Win32 API on which all of the cmdlets in PowerShell are based.

Usage: $short_path = Shorten-Path “some-long-path” “temporary-directory-path”

What the function will do is simply make a junction point in the temporary directory, and return a shorter path that points to the same place as the original long path, but through the junction point. This is only if the path is too long: if not, then it will return the original path.

# Returns a shortened path made with junctions to circumvent 260 path length in win32 API and so PowerShell
function Shorten-Path {
	[CmdletBinding()]
	param( 
		[Parameter(Mandatory=$true,
				   Position=0,
				   ValueFromPipeline=$true,
				   HelpMessage="Path to shorten.")]
		[string]$Path,
		[Parameter(Mandatory=$true,
				   Position=1,
				   ValueFromPipeline=$false,
				   HelpMessage="Path to existing temp directory.")]
		[string][ValidateScript({Test-Path -LiteralPath $_ -PathType Container})]$TempPath    
	)
 
	begin {
		# Requirements check
		if (-not $script:junction) {$script:junction = @{};}
		$max_length = 248; # this is directory max length; for files it is 260.
	}
 
	process {
		# First check whether the path must be shortened.
		# Write-Warning "$($path.length): $path"
		if ($Path.length -lt $max_length) {
			Write-Debug "Path length: $($Path.length) chars."; 
			return $Path;
		}
 
		# Check if there is allready a suitable symlink	
		$path_sub = $junction.keys | foreach { if ($Path -Like "$_*") {$_} } | Sort-Object -Descending -Property length | Select-Object -First 1;
		if ($path_sub) {
			$path_proposed = $Path -Replace [Regex]::Escape($path_sub), $junction[$path_sub];
			if ($path_proposed.length -lt $max_length) {
				# assert { Test-Path $junction[$path_sub] } "Assertion failed in junction path check $($junction[$path_sub]) for path $path_sub.";
				return $path_proposed;
			}
		}
 
		# No suitable symlink so make new one and update junction
		$path_symlink_length = ($TempPath + '\' + "xxxxxxxx").length;
		$path_sub = ""; # Because it is allready used in the upper half, and if it is not empty, we get nice errors...
		$path_relative = $Path;
		# Explanation: the whole directory ($Path) is taken, and with each iteration, a directory node is taken from
		# $path_relative and put in $path_sub. This is done until there is nothing left in $path_relative.
		while ($path_relative -Match '([\\]{0,2}[^\\]{1,})(\\.{1,})') {
			$path_sub += $matches[1];
			$path_relative = $matches[2];
			if ( ($path_symlink_length + $path_relative.length) -lt $max_length ) {
				$tmp_junction_name = $TempPath + '\' + [Convert]::ToString($path_sub.gethashcode(), 16);
				# $path_sub might be very large. We can not link to a too long path. So we also need to shorten it (i.e. recurse).
				$mklink_output = cmd /c mklink /D """$tmp_junction_name""" """$(Shorten-Path $path_sub $TempPath)""" 2>&1;
				$junction[$path_sub] = $tmp_junction_name;
				# assert { $LASTEXITCODE -eq 0 } "Making link $($junction[$path_sub]) for long path $path_sub failed with ERROR: $mklink_output.";
				return $junction[$path_sub] + $path_relative;
			}
		}
 
		# Path can not be shortened...
		# assert $False "Path $path_relative could not be shortened. Check code!"
	} 
 
	end {}
}

In $junction variable, all the junction points are stored, so you can remove them afterwards with:

foreach ($link in $junction.values) {
	$rmdir_error = cmd /c rmdir /q """$link""" 2>&1;
	if ( $LASTEXITCODE -ne 0 ) { Write-Warning "Removing link $link failed with ERROR: $rmdir_error." };
}
$junction.clear();

I am also using the assert function which you can find here.

03/04/2013

Search for methods and properties in WMI with PowerShell

Filed under: PowerShell,Programming — Tags: , — Davor @ 10:14

Here is some simple recursive code to search for methods/properties in the WMI.

Much can be improved, but for now, it is a start.

##########################################################################################
## CHANGE LOG
#########################
##
## v0.1
## - first version
##########################################################################################
 
filter seek_properties {
	$_ | Get-Member | select-object name | property $_;
}
 
filter property ($obj) {			
			if ((++$global:i % 100000) -eq 0) { echo $i;}
			if ($obj.($_.name) -match 'YOUR SEARCH STRING') {
				echo "Object $($obj.__path) contains in property $($_.name): $($obj.$($_.name))";
				echo $obj;
				echo '-----------------------------------------------';
			}
}
 
function start_search ($namespace) {
	Get-WmiObject -Namespace $namespace -list * | foreach {
		if ($_.__CLASS -eq "__NAMESPACE") {
			Get-WmiObject -Namespace $namespace $_.__CLASS | foreach { start_search "$($_.__NAMESPACE)\$($_.Name)"; } 		
		} else {
			echo "$namespace`: $($_.__CLASS)";
			Get-WmiObject -Namespace $namespace $_.__CLASS | seek_properties; 
		}
	}
 
}
 
$global:i = 0;
 
start_search "ROOT";

This script can take quite some time to finish…

28/07/2012

Transcode mkv – batch file for windows

Filed under: batch,Programming — Tags: , , , , , — Davor @ 10:47

What it does: transcode your mkv to x264 with a specified file size. You drag & drop your mkv-file on the batch file, chose some options like target size, and it starts the transcoding process. It re-encodes the video and leaves all other streams intact (unless you exclude them). It uses handbreakcli and mkvtoolnix.

I based this mainly on this fine article (for linux).

It’s still a work in progress, but currently, it does the job.

What you need is:

Make sure you set the paths in the beginning of the script. The rest should work.

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: CHANGE LOG
:::::::::::::::::::::::::
:: v0.8.8
:: - added priority option to :HANDBRAKE function
:: - mkvmerge priority levels set slightly higher
:: 
:: v0.8.7
:: - added user choice for destination directory
:: - added mkv_transcode_settings.ini file
::
:: v0.8.6
:: - compatible with handbrake v0.9.9 and "--h264-level" option: lower handbrakecli versions are not supported!
::
:: v0.8.3
:: - fixed bug with stream selection when there is no stream name.
:: - Denoise: off - Decomb: on printing bug fixed.
::
:: v0.8.3
:: - fixed auto resolution to choose correct number of ref frames based on source resolution
::
:: v0.8.2
:: - decomb encoding option added
:: - decomb option added in analysis
:: - analysis code put into a function
::
:: v0.8.1
:: - added decomb option for testing
::
:: v0.8.0
:: - added 1/3 of a BluRay50 size
:: - fixed bug that overwrites the original when demuxing without subs
::
:: v0.7.9
:: - started using tee from UnxUtils
:: - quick denoising option during analysis, based on a little statistical analysis
::
:: v0.7.8
:: - ffmpeg now uses some default options %ffmpeg_options%
:: - cleanup temporary disabled due to casual Error 1450 
::
:: v0.7.7
:: - sample duration reduced to 5 sec from 10 sec
:: - extra error print when final mkvmerge fails
::
:: v0.7.6
:: - added denoising options in extended analysis
::
:: v0.7.5
:: - conversion_map improved
:: - bug fixed where channels wasn't resolved in conversion_map
::
:: v0.7.4
:: - bug fixed where extended analysis on 1280pixels width didn't try DVD resolution
:: - removed auto-size calculation
:: - audio track encoding shows a bit more info
::
:: v0.7.3
:: - the code for neglecting audio and/or subs is now put in 1 function
::
:: v0.7.2
:: - extended analysis makes predictions on size based conversion_map
:: - make_samples funtion can now work silently
:: - added a few smaller auto sizes
:: - evaluate function now works with setlocal 1033 and doesn't replace "," with "."
:: - prints filename and versions when starting up
::
:: v0.7.1
:: - extended analysis option
:: - large procedure for auto size calculation split into functions
::
:: v0.7
:: - handbrakecli is now run from within a function
:: - evaluate.bat is now a function
::
:: v0.6.3
:: - SETLOCAL EnableDelayedExpansion standard on
:: - MP3 max quality VBR encoding added
:: - filter for ffmpeg and mkvmerge because of wtee added
::
:: v0.6.2
:: - use ffmpeg for demuxing
::
:: v0.6.1
:: - added sound processing per track
::
:: v0.6.0
:: - added logging: batch calls itself and pipes all through wtee
::
:: v0.5.7
:: - new tmp folder
:: - orig_rest_size is now 0 if non existant
:: - sample pics currently disabled - ffmpeg crashes sometimes
:: - generates unique final name for mkv if one allready exists
:: - removed vedit
:: - added wtee
::
:: v0.5.6
:: - picture sample names and seek changed
::
:: v0.5.5
:: - added checks for neglected audio and subtitle tracks
:: - bug resolved: doesn't anymore stop when there are no subtitles
:: - added timestamps
::
:: v0.5.4
:: - resolved crashing when there is no tmp_orig_rest or tmp_orig_audio
::
:: v0.5.3
:: - added default AC-3 bitrate (ffmpeg choses)
:: - added denoise option
::
:: v0.5.2
:: - end comparison of original and transcode added
:: - now makes sample pictures of original and transcode
::
:: v0.5.1
:: - track elimination happens now during demuxing
:: - simple error (10%) for autosize bitrate prediction introduced
::
:: v0.5
:: - audio conversion support
::
:: v0.4.3
:: - changed samples from 3 to 6
:: - decreased sample length from 20s to 10s
::
:: v0.4.2
:: - added second way to determine sample bitrate
:: - added simple mkvinfo error handling
:: - double duration calculation removed
::
:: v0.4.1
:: - auto size RF lowered to 20 because of visible color glitches in samples
:: - added simple error handling to handbrakecli
:: - DVD resolution is now --maxWidth 720 instead of --width 720
::
:: v0.4
:: - added auto size option based on bitrate predicted by RF 21
::
:: v0.3.7
:: - added extra sizes: 1.37GiB & 2.05GiB
::
:: v0.3.6
:: - exclude tracks from the final output
:: - added simple error handling for mkvnixtools
::
:: v0.3.5
:: - black & white encoding
::
:: v0.3.4
:: - added 700MiB size
::
:: v0.3.3
:: - added affinity
::
:: v0.3.2
:: - custom crop now supports 4 input values
::
:: v0.3.1
:: - added custom crop
::
:: v0.3
:: - added choice: preset
:: - added choice for quality based encoding
:: - added suffix based on selected settings
::
:: v0.2.6
:: - now choses handbrakeCLI based on x86 or x64
::
:: v0.2.5
:: - now works with mkvnixtool v5.5.0
:: - option for bitrate added
::
:: v0.2.4
:: - added choice: resolution, targetsize
::
:: v0.2.3
:: - added choice: crop
:: - added error handling in case there is no input argument
::
:: v0.2.2
:: - now works with drag/drop
::
:: v0.1
:: - first version
::
:: TODO: Improve logging
:: BUG: Because under wtee all is local, the copied log has not the right (possible) suffix number
:: BUG: @ECHO Encode settings code: %suffix% >> %log_file% --> wtee is accessing the log, so you can't write in it. And after it's done, all the vars are deleted, so this can not be used.
:: BUG: some redirects are added to handbrakecli and mkvmerge because wtee has issues showing progress
:: TODO: Work with avi
:: TODO: Also show the size if -% introduces an other size. 
:: TODO: Warning: matroska_reader: Could not keep the track UID 1 because it is already allocated for the new file. Waarschijnlijk omdat subs mij audio en rest steken, etc.
:: TODO: Temporarily disabled clean up to fix: Error: Could not write to the output file: 1450 (Insufficient system resources exist to complete the requested service.) Error: 0 -- This is currently worked around by first trying to mux the file to local dir in %destination_folder%.
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
 
@IF "%~1" EQU "" GOTO ERROR_NO_ARG
@IF NOT EXIST %1 GOTO ERROR_BAD_ARG
 
@SET input="%~1"
@SET input_name=%~n1
 
@SET tmp_folder=W:\Temp\Scripts\Transcode\%input_name%
@SET "usersettings=%~dp0mkv_transcode_settings.ini"
@IF NOT EXIST "%tmp_folder%" @MKDIR "%tmp_folder%"
@IF NOT EXIST "%usersettings%" echo. 2>"%usersettings%"
 
@SET log_file="%tmp_folder%\%input_name%.encode.log"

:: Logging mechanism is activated here.
:: The batch checks whether 2501 is put as second arg. If not, it
:: re-executes with piping to wtee.
:: One can omit this (i.e. logging) by putting 2501 as the second argument.
:: Using wtee makes all the variables in the called batch local!
@IF "%2" NEQ "2501" @CALL %0 %1 2501 | W:\tools\unxutils\v2007\usr\local\wbin\tee.exe %log_file% & GOTO MOVE_LOG

::@SETLOCAL
::@SETLOCAL EnableExtensions
@SETLOCAL EnableDelayedExpansion
 
@ECHO.
:: Setting up PATH variables
@CALL W:\Tools\handbrake\set_path.bat
@CALL W:\Tools\ffmpeg\set_path.bat
 
@SET filter=FINDSTR /I /V "progress: size= configuration: copyright libav libsw libpost [q]"
@SET ffmpeg_options=-nostats -y -analyzeduration 100M -probesize 100M
 
@SET mkvtoolnix=W:\Tools\mkvtoolnix\v5.5.0
@SET path=%path%;%mkvtoolnix%;%~dp0
 
@SET tmp_orig_video="%tmp_folder%\tmp_orig_video.mkv"
@SET tmp_orig_audio="%tmp_folder%\tmp_orig_audio.mkv"
@SET tmp_orig_rest="%tmp_folder%\tmp_orig_rest.mkv"
@SET tmp_enc_audio="%tmp_folder%\tmp_enc_audio.mkv"
@SET tmp_enc_audio_and_rest="%tmp_folder%\tmp_enc_audio_and_rest.mkv"
@SET tmp_enc_video="%tmp_folder%\tmp_enc_video.mkv"
@SET tmp_sample_bitrates_log="%tmp_folder%\tmp_sample_bitrates.txt"
@SET tmp_sample_activity_log="%tmp_folder%\tmp_sample_activity_log.txt"
@SET tmp_orig_mkvinfo="%tmp_folder%\MKVINFO_ORIGINAL.txt"
@SET tmp_enc_mkvinfo="%tmp_folder%\MKVINFO_ENCODE.txt"
@SET tmp_orig_identify_ffmpeg="%tmp_folder%\FFMPEG_ID_ORIGINAL.txt"
@SET tmp_orig_identify_handbrake="%tmp_folder%\HANDBRAKE_SCAN_ORIGINAL.txt"
@SET tmp_pre-enc_identify_ffmpeg="%tmp_folder%\FFMPEG_ID_PRE-ENCODE.txt"
@SET tmp_identify="%tmp_folder%\tmp_mkvmerge_identity.txt"
 
@SET CD-28P80min=26214400
@SET CD-14P80min=52428800
@SET CD-7P80min=104857600
@SET CD-2H80min=183500000
@SET CD-H80min=367000000
@SET CD-80min=734000000
@SET CD-80minX2=1468000000
@SET CD-80minX3=2202000000
@SET DVD-SL=4690000000
@SET DVD-DL=8535000000
@SET Bluray-HSL=12505000000
@SET Bluray-TDL=16664473110
@SET Bluray-SL=25000000000
@SET Bluray-DL=50000000000
 
@SET cmd_resolution_1080=--loose-anamorphic --width 1920 --maxHeight 1080
@SET cmd_resolution_720=--loose-anamorphic --width 1280 --maxHeight 720
@SET cmd_resolution_576=--loose-anamorphic --maxWidth 720 --maxHeight 576
@SET cmd_resolution_auto=--strict-anamorphic

:: Conversionmap MUST be set with DisabledDelayedExpansion!
:: For possible !keys!, see :MAKE_AUDIO_MAP
:: Here are some: bitrate, sampleres, channels, frequency, codec
@SETLOCAL DisableDelayedExpansion
@SET conversion_map=^
#InStr(1,""!codec!"",""DTS"",1)^>0 AND InStr(1,""!channels!"",""5.1"",1)^>0 AND !bitrate!^>768:codec--ac3;bitrate--640#^
#InStr(1,""!codec!"",""PCM"",1)^>0 AND ( InStr(1,""!channels!"",""mono"",1)^>0 OR InStr(1,""!channels!"",""stereo"",1)^>0 ) AND !bitrate!^>=1152:codec--mp3;bitrate--192#
@SETLOCAL EnableDelayedExpansion
 
@SET quality_based_encoding_rf=20
@SET auto_size_rf=20
@SET nr_samples_for_autosize=6
@SET sample_duration_sec=10
@SET percentage_autosize_bitrate_error=15
@SET nr_picture_samples=5
 
@CALL :EVALUATE "chr(8)"
@SET bs_char=%result%
:: The following three lines escape a new line and put it in a variable for later.
:: The 2 empty lines are important!
@SET newline=^
 
 
@ECHO.
@ECHO Version: %~n0
@ECHO ---------------------------------------------------------------------------
@ECHO Started working on %input_name% 
@ECHO Timestamp: %date% %time%
@ECHO ---------------------------------------------------------------------------
 
@ECHO.
@CHOICE /C "1234" /N /T 0 /D 4 /M "# cores to work with: 1, 2, 3, 4?"
@SET choice_affinity=%ERRORLEVEL%
@IF %choice_affinity% EQU 1 SET affinity=1
@IF %choice_affinity% EQU 2 SET affinity=3
@IF %choice_affinity% EQU 3 SET affinity=7
@IF %choice_affinity% EQU 4 SET affinity=F 
 
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO Analysing container...
@ECHO ---------------------------------------------------------------------------
 
@mkvinfo --ui-language English %input% > %tmp_orig_mkvinfo%
@IF %ERRORLEVEL% NEQ 0 GOTO ERROR_MKVINFO
 
@CALL :HANDBRAKE %input%, "", "" , "", "--previews 30:0 --scan", "NORMAL" 1>&2 2> %tmp_orig_identify_handbrake%

:: Calculate size
@FOR /F "tokens=*" %%A IN (%input%) DO @SET input_size=%%~zA
@CALL :EVALUATE "FormatNumber(%input_size%/1024/1024)"
@ECHO File size: %result% MiB

:: Calculate duration
@FOR /F "tokens=4" %%A IN ('FIND "Duration:" ^< %tmp_orig_mkvinfo%') DO @SET duration_sec=%%A
@IF "%duration_sec%"=="" GOTO ERROR_NO_DURATION
@SET duration_sec=%duration_sec:~0,-1%

:: Show duration (duration_sec isn't used because it is in seconds, this one is in HMS)
:: @FOR /F "tokens=2 delims=, " %%D IN ('TYPE %tmp_orig_identify_handbrake% ^|FIND /I "  Duration: "') DO @ECHO Video duration: %%D HMS
@FOR /F "tokens=2 delims=()" %%D IN ('TYPE %tmp_orig_mkvinfo% ^| FIND /I "| + Duration:"') DO @ECHO Video duration: %%D HMS

:: Show resolution:
@FOR /F "tokens=5 delims= " %%W IN ('TYPE %tmp_orig_mkvinfo% ^| FIND /I "|   + Pixel width:"') DO @SET orig_pixel_width=%%W
@CALL :EVALUATE "IsNumeric(%orig_pixel_width%)"
@IF %result% NEQ True @ECHO %orig_pixel_width% is not numeric. & GOTO QUIT 
@FOR /F "tokens=5 delims= " %%H IN ('TYPE %tmp_orig_mkvinfo% ^| FIND /I "|   + Pixel height:"') DO @SET orig_pixel_height=%%H
@CALL :EVALUATE "IsNumeric(%orig_pixel_height%)"
@IF %result% NEQ True @ECHO %orig_pixel_height% is not numeric. & GOTO QUIT 
@ECHO Resolution: %orig_pixel_width%x%orig_pixel_height%

:: Show auto crop values
@FOR /F "tokens=3" %%C IN ('TYPE %tmp_orig_identify_handbrake% ^| FIND /I "autocrop:"') DO @SET "autocrop=%%C" & SET autocrop=!autocrop:/=:!
@ECHO Auto crop: %autocrop%
 
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO Setting destination folder...
@ECHO ---------------------------------------------------------------------------

:: Here we show all the destination folders from the user settings file and let hims specify one or write a new one
@SET /A destpathcounter=0
@ECHO.
@FOR /F "tokens=2 delims==" %%S IN ('TYPE %usersettings% ^| FIND /I "destinationpath="') DO @(
	@IF EXIST %%S @(
		@ECHO   [!destpathcounter!] %%S
		@SET "destpathchoice!destpathcounter!=%%S"
		@SET /A destpathcounter+=1
	)
)
@ECHO.
:SET_DESTINATION_FOLDER
@SET /P destination_folder=Destination (number or path): 

:: Check first if a number is given...
@ECHO %destination_folder%|FINDSTR /R "[^0-9]" > NUL
@IF %errorlevel% EQU 0 @( REM contains non-numeric characters, so it is a path
	@IF NOT EXIST "%destination_folder%" @( 
		@ECHO Specified folder doesn't exist. Try again...
		@ECHO.
		@GOTO SET_DESTINATION_FOLDER
	) ELSE @( REM Folder exists: should be set in user settings file if not allready there, and script continued.
		@FOR /F %%C IN ('TYPE %usersettings% ^| FIND /I /C "%destination_folder%"') DO @(
			@IF "%%C" EQU "0" @( REM %destination_folder% not found in user settings file, so it can be added
				@ECHO destinationpath=%destination_folder%>> %usersettings%
			)
		)
	)
) ELSE @( REM %destination_folder% contains a number
	@IF NOT DEFINED destpathchoice%destination_folder% @( 
		@ECHO Invalid number. Try again.
		@GOTO SET_DESTINATION_FOLDER
	) ELSE @(
		SET destination_folder=!destpathchoice%destination_folder%!
	)
)
 
@ECHO Destination folder chosen: %destination_folder%
 
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO Choosing audio and subtitle tracks...
@ECHO ---------------------------------------------------------------------------

:: Show stream info
@ECHO.
@ffmpeg %ffmpeg_options% -i %input% 2>&1 | @FINDSTR /I /R "#0[.:]" > %tmp_orig_identify_ffmpeg%
@IF %ERRORLEVEL% NEQ 0 GOTO QUIT
@TYPE %tmp_orig_identify_ffmpeg% | @FINDSTR /I /R "#0[:].*Audio: Subtitle:"
 
@COPY /Y %tmp_orig_identify_ffmpeg% %tmp_pre-enc_identify_ffmpeg% > NUL

:: Choose audio tracks to neglect
@ECHO.
@SET /P choice_naudio=Neglect audio streams (comma delimited)?: 
@CALL :NEGLECT_TRACK_CMDLINE "Audio", "%choice_naudio%", str_neglect_audio_tracks, error_code
@IF "%error_code%" NEQ "0" @ECHO Aborting... & @GOTO CLEAN_UP

:: Check whether there are any more tracks
@FOR /F %%C IN ('TYPE %tmp_pre-enc_identify_ffmpeg% ^| FIND /I /C "Audio:"') DO @(
	@IF "%%C" EQU "0" @( 
		@ECHO No audio tracks left to mux... Aborting. & @GOTO CLEAN_UP 
	)
)

:: Choose subtitle tracks to neglect
@ECHO.
@SET /P choice_nsubtitle=Neglect subtitle streams (comma delimited)?: 
@CALL :NEGLECT_TRACK_CMDLINE "Subtitle", "%choice_nsubtitle%", str_neglect_subtitle_tracks, error_code
@IF "%error_code%" NEQ "0" @ECHO Aborting... & @GOTO CLEAN_UP

:: Check whether there are any more tracks --> if not, set var to inform there will be no "rest"
@FOR /F %%C IN ('TYPE %tmp_pre-enc_identify_ffmpeg% ^| FIND /I /C "Subtitle:"') DO @( 
	@IF "%%C" NEQ "0" @(
		SET there_exist_subs=true
	)
)

:: Now make audio map
@CALL :MAKE_AUDIO_MAP audio_map, %tmp_pre-enc_identify_ffmpeg%
::@ECHO %audio_map%

:: Get number of audio streams
@CALL :SEEK_IN_MAP streams, "!audio_map!", nr_audio_streams
@SET /A nr_max_stream=%nr_audio_streams%-1
 
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO Setting general options...
@ECHO ---------------------------------------------------------------------------

:: Select x264 preset
@ECHO.
@CHOICE /C "PVSFU" /N /T 120 /D v /M "x264 preset: [P]lacebo, [V]eryslow, [S]low, [F]ast, [U]ltrafast?"
@SET x-preset=%ERRORLEVEL%
@IF %x-preset% EQU 1 SET useropt=%useropt% --x264-preset placebo & SET suffix=%suffix%-placebo
@IF %x-preset% EQU 2 SET useropt=%useropt% --x264-preset veryslow & SET suffix=%suffix%-veryslow
@IF %x-preset% EQU 3 SET useropt=%useropt% --x264-preset slow & SET suffix=%suffix%-slow
@IF %x-preset% EQU 4 SET useropt=%useropt% --x264-preset fast & SET suffix=%suffix%-fast
@IF %x-preset% EQU 5 SET useropt=%useropt% --x264-preset ultrafast & SET suffix=%suffix%-ultrafast

:: Cropping: Top:Bottom:Left:Right
@ECHO.
@CHOICE /C "ANC" /N /T 120 /D A /M "Crop [A]uto (%autocrop%), [N]one, [C]ustom?"
@SET crop=%ERRORLEVEL%
@IF %crop% EQU 1 SET useropt=%useropt% --crop %autocrop% & SET suffix=%suffix%-A%!!%Crop
@IF %crop% EQU 2 SET useropt=%useropt% --crop 0:0:0:0 & SET suffix=%suffix%-NoCrop
@IF %crop% EQU 3 (
@SET /P crop_top=Top:
@SET /P crop_bottom=Bottom:
@SET /P crop_left=Left:
@SET /P crop_right=Right:
@SET useropt=%useropt% --crop !crop_top!:!crop_bottom!:!crop_left!:!crop_right! 
@SET suffix=%suffix%-C!crop_top!;!crop_bottom!;!crop_left!;!crop_right!Crop
)

:: Source grayscale?
@ECHO.
@CHOICE /C "YN" /N /T 120 /D N /M "Movie black & white? [Y/N]"
@SET bw_movie=%ERRORLEVEL%
@IF %bw_movie% EQU 1 SET useropt=%useropt% --grayscale & SET suffix=%suffix%-bw
@IF %bw_movie% EQU 2 ECHO Do Nothing > NUL

:: Shall we do an extended analysis?
@ECHO.
@CHOICE /C "YN" /N /T 60 /D Y /M "Extended analysis? [Y/N]"
@SET choice_ext_analysis=%ERRORLEVEL%
@IF %choice_ext_analysis% EQU 1 ECHO Do Nothing > NUL
@IF %choice_ext_analysis% EQU 2 GOTO SKIP_EXTENDED_ANALYSIS

:: What kind of denoise analysis shall we do?
@CHOICE /C "NQF" /N /T 60 /D N /M " - Denoise analysis: [N]one, [Q]uick or [F]ull?"
@SET choice_denoise_analysis=%ERRORLEVEL%
@IF %choice_denoise_analysis% EQU 1 ECHO Do Nothing > NUL
@IF %choice_denoise_analysis% EQU 2 ECHO Do Nothing > NUL
@IF %choice_denoise_analysis% EQU 3 ECHO Do Nothing > NUL

:: Decomb analysis?
@CHOICE /C "NY" /N /T 60 /D N /M " - Decomb analysis: [Y/N]?"
@SET choice_decomb_analysis=%ERRORLEVEL%
@IF %choice_decomb_analysis% EQU 1 ECHO Do Nothing > NUL
@IF %choice_decomb_analysis% EQU 2 ECHO Do Nothing > NUL
 
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO Extended analysis...
@ECHO ---------------------------------------------------------------------------

:: Here we update the audio_map with new possible encodings through conversion_map.
:: In the audio_map, the original track is always designated as tracknr.0
:: Tracknr.x, where x > 0, are all the other possibilities for encoding.
@FOR /L %%T IN (0,1,%nr_max_stream%) DO (
	@SET conv_nr=0
	@CALL :SEEK_IN_MAP codec%%T.0, "!audio_map!", codec
	@CALL :SEEK_IN_MAP bitrate%%T.0, "!audio_map!", bitrate
	@CALL :SEEK_IN_MAP channels%%T.0, "!audio_map!", channels
	@CALL :SEEK_IN_MAP frequency%%T.0, "!audio_map!", frequency
	@CALL :SEEK_IN_MAP sampleres%%T.0, "!audio_map!", sampleres
	@FOR /F "tokens=1 delims=#" %%i in ("%conversion_map:#=!newline!%") do @(
		@CALL :AUDIO_MAP_UPDATE "%%i", %%T, conv_nr, audio_map
	)
	@SET audio_map=!audio_map!;conversions%%T--!conv_nr!
)

:: Now we make a map for the possible conversions
:: This map is put into %scramble%
CALL :SCRAMBLE
::@ECHO "SCRAMBLE: %scramble%"

:: Test original pixel-by-pixel compression
@ECHO.
@ECHO Analysis on original (%orig_pixel_width%x%orig_pixel_height%) resolution.
@ECHO -----------------------
@CALL :PRINT_ANALYSIS "%orig_pixel_height%p(source)", "%cmd_resolution_auto%"

:: There is no valid case in which 1080p compression should be tried.
:: @CALL :MAKE_SAMPLES %input%, "tmp_video_sample_1080p", "%useropt% %cmd_resolution_1080%", avg_bitrate

:: Test 720p compression: (1) if source is HD 1080 --> try subsample
@CALL :EVALUATE "%orig_pixel_height%>720 OR %orig_pixel_width%>1280"
@IF %result% EQU True @(
	@ECHO.
	@ECHO Analysis on subsampled HD 720p resolution.
	@ECHO -----------------------
	@CALL :PRINT_ANALYSIS "720p", "%cmd_resolution_720%"
)

:: Test DVD compression: (1) if source is HD --> try subsample
@CALL :EVALUATE "%orig_pixel_height%>576 OR %orig_pixel_width%>720"
@IF %result% EQU True @(
	@ECHO.
	@ECHO Analysis on subsampled DVD resolution
	@ECHO -----------------------
	@CALL :PRINT_ANALYSIS "576p", "%cmd_resolution_576%"
)
 
@ECHO.
@PAUSE
 
:SKIP_EXTENDED_ANALYSIS
 
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO Setting options...
@ECHO ---------------------------------------------------------------------------
 
@ECHO.
@CHOICE /C "SQB" /N /T 120 /D S /M "Bitrate: [S]ize, [Q]uality, [B]itrate?"
@SET choice_bitrate=%ERRORLEVEL%
@IF %choice_bitrate% EQU 1 @ECHO Do Nothing > NUL
@IF %choice_bitrate% EQU 2 SET bitrate_control=--quality %quality_based_encoding_rf% & SET suffix=q20
@IF %choice_bitrate% EQU 3 @( 
	SET /P new_bitrate=New Bitrate: 
	IF "%new_bitrate%" NEQ "" @( ECHO New bitrate set to %new_bitrate% kbit/s & SET suffix=b!new_bitrate!)
	ELSE @( ECHO No bitrate set... aborting. & @GOTO CLEAN_UP )
)
::@IF %choice_bitrate% EQU 4 SET suffix=AutoSize 

:: Sound choices
:: -map 0:!audio_track_nr! is used for the %tmp_orig_audio%, which has not necessarily the same ID's as the %input%
@ECHO.
@FOR /L %%T IN (0,1,%nr_max_stream%) DO @(
	@CALL :SEEK_IN_MAP codec%%T.0, "%audio_map%", codec
	@CALL :SEEK_IN_MAP bitrate%%T.0, "%audio_map%", bitrate
	@CHOICE /C "PADM" /N /T 120 /D P /M "Audio track number %%T [!codec! ^@!bitrate!kbit/s]:!newline!	--^> [P]ass-through, [A]C-3 640kb/s, AC-3 [D]efault, [M]P3?"
	@SET choice_audio=!ERRORLEVEL!
	@IF !choice_audio! EQU 1 SET audio_enc=!audio_enc! -map 0:a:%%T -c:a:%%T copy & SET suffix=!suffix!-A%%Tpt
	@IF !choice_audio! EQU 2 SET audio_enc=!audio_enc! -map 0:a:%%T -c:a:%%T ac3 -b:a:%%T 640k & SET suffix=!suffix!-A%%Tac3Max
	@IF !choice_audio! EQU 3 SET audio_enc=!audio_enc! -map 0:a:%%T -c:a:%%T ac3 & SET suffix=!suffix!-A%%Tac3Def
	@IF !choice_audio! EQU 4 SET audio_enc=!audio_enc! -map 0:a:%%T -c:a:%%T libmp3lame -q:a:%%T 0 -compression_level 0 -id3v2_version 3 & SET suffix=!suffix!-A%%Tmp3
)
 
@ECHO.
@CHOICE /C "A17D" /N /T 120 /D A /M "Resolution [A]uto, [1]080, [7]20, [D]VD?"
@SET res=%ERRORLEVEL%
@IF %res% EQU 1 SET useropt=%useropt% %cmd_resolution_auto% & SET suffix=%suffix%-ResStrict
@IF %res% EQU 2 SET useropt=%useropt% %cmd_resolution_1080% & SET suffix=%suffix%-ResL1080p
@IF %res% EQU 3 SET useropt=%useropt% %cmd_resolution_720% & SET suffix=%suffix%-ResL720p
@IF %res% EQU 4 SET useropt=%useropt% %cmd_resolution_576% & SET suffix=%suffix%-ResL576p
 
@ECHO.
@CHOICE /C "NWMS" /N /T 120 /D N /M "Noise reduction [N]one, [W]eak, [M]edium, [S]trong?"
@SET denoise=%ERRORLEVEL%
@IF %denoise% EQU 1 @ECHO Do Nothing > NUL
@IF %denoise% EQU 2 SET useropt=%useropt% --denoise=weak & SET suffix=%suffix%-denW
@IF %denoise% EQU 3 SET useropt=%useropt% --denoise=medium & SET suffix=%suffix%-denM
@IF %denoise% EQU 4 SET useropt=%useropt% --denoise=strong & SET suffix=%suffix%-denS
 
@ECHO.
@CHOICE /C "NY" /N /T 120 /D N /M "Decomb [Y]es or [N]o?"
@SET choice_decomb=%ERRORLEVEL%
@IF %choice_decomb% EQU 1 @ECHO Do Nothing > NUL
@IF %choice_decomb% EQU 2 SET useropt=%useropt% --decomb & SET suffix=%suffix%-decomb
 
@IF %choice_bitrate% NEQ 1 GOTO SKIP_SIZE_SET
@ECHO.
@CHOICE /C "123456789" /N /T 120 /D 4 /M "Targetsize 700MiB [1], 1.37GiB [2], 2.05GiB [3], 4.37GiB [4], 7.95GiB [5], 12.5GB [6], 15.5GB [7]?"
@SET size=%ERRORLEVEL%
@IF %size% EQU 1 SET targetsize=%CD-80min%
@IF %size% EQU 2 SET targetsize=%CD-80minX2%
@IF %size% EQU 3 SET targetsize=%CD-80minX3%
@IF %size% EQU 4 SET targetsize=%DVD-SL%
@IF %size% EQU 5 SET targetsize=%DVD-DL%
@IF %size% EQU 6 SET targetsize=%Bluray-HSL%
@IF %size% EQU 7 SET targetsize=%Bluray-TDL%
@IF %size% EQU 8 SET targetsize=%Bluray-SL%
@IF %size% EQU 9 SET targetsize=%Bluray-DL%
:SKIP_SIZE_SET

::GOTO SKIP_DEMUX
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO De-muxing...
@ECHO ---------------------------------------------------------------------------

:: here we make tmp_orig_video, tmp_orig_audio, tmp_orig_rest which we use later on!
:: here we also substract the tracks that are to be excluded.

:: mkvextract doesn't seem to extract meta-data? Handbrake will not work with it's video stream.
:: mkvextract --ui-language English tracks %input% 0:%tmp_orig_video% 1:%tmp_orig_audio%
 
@IF "%there_exist_subs%" EQU "true" (
@START /B /LOW /AFFINITY %affinity% /WAIT ffmpeg %ffmpeg_options% -i %input% -map 0:v -c:v copy %tmp_orig_video% -map 0:a -c:a copy %str_neglect_audio_tracks% %tmp_orig_audio% -map 0 -vn -an -c:s copy %str_neglect_subtitle_tracks% %tmp_orig_rest% 2>&1
) ELSE (
@START /B /LOW /AFFINITY %affinity% /WAIT ffmpeg %ffmpeg_options% -i %input% -map 0:v -c:v copy %tmp_orig_video% -map 0:a -c:a copy %str_neglect_audio_tracks% %tmp_orig_audio% 2>&1 
)
@IF %ERRORLEVEL% NEQ 0 GOTO CLEAN_UP
:SKIP_DEMUX

:: storing sizes for printing later
@FOR /F "tokens=*" %%A IN (%tmp_orig_video%) DO @SET orig_video_size=%%~zA
@FOR /F "tokens=*" %%A IN (%tmp_orig_audio%) DO @SET orig_audio_size=%%~zA
@FOR /F "tokens=*" %%A IN (%tmp_orig_rest%) DO @SET orig_rest_size=%%~zA
@IF "%orig_rest_size%" EQU "" SET orig_rest_size=0
 
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO Transcoding audio...
@ECHO ---------------------------------------------------------------------------

:: ffmpeg outputs everything through stderr (2)
::@IF %audio_enc% EQU copy ECHO Passing audio through... & MOVE /Y %tmp_orig_audio% %tmp_enc_audio% > NUL
@START /B /LOW /AFFINITY %affinity% /WAIT ffmpeg %ffmpeg_options% -i %tmp_orig_audio% %audio_enc% %tmp_enc_audio% 2>&1 & @IF %ERRORLEVEL% NEQ 0 GOTO CLEAN_UP
:SKIP_AUDIO_TRANSCODING

:: for printing info later
@FOR /F "tokens=*" %%A IN (%tmp_enc_audio%) DO @SET enc_audio_size=%%~zA
 
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO Muxing audio with rest...
@ECHO ---------------------------------------------------------------------------

:: this is needed so one can calculate the bitrate of the video for a given size.
@IF "%there_exist_subs%" EQU "true" (
@mkvmerge --priority lower --ui-language English -o %tmp_enc_audio_and_rest% %tmp_enc_audio% %tmp_orig_rest% | %filter%
@IF %ERRORLEVEL% EQU 2 GOTO CLEAN_UP
) ELSE (
@ECHO There is only audio...
@MOVE %tmp_enc_audio% %tmp_enc_audio_and_rest% > NUL
)

:: needed for calculations
@FOR /F "tokens=*" %%A IN (%tmp_enc_audio_and_rest%) DO @SET enc_audio_and_rest_size=%%~zA

REM @IF %choice_bitrate% NEQ 4 GOTO SKIP_AUTO_SIZE_CALCULATION 
REM @ECHO.
REM @ECHO ---------------------------------------------------------------------------
REM @ECHO Calculating auto target size...
REM @ECHO ---------------------------------------------------------------------------

REM @CALL :MAKE_SAMPLES %tmp_orig_video%, "tmp_video_sample", "%useropt%", "Y", autosize_predicted_bitrate

REM @CALL :EVALUATE "CInt(%autosize_predicted_bitrate%*%percentage_autosize_bitrate_error%/100)"
REM @SET autosize_predicted_bitrate_error=%result%
REM @ECHO Predicted bitrate for %auto_size_rf% RF set to %autosize_predicted_bitrate% kbit/s (+/-%autosize_predicted_bitrate_error% kbit/s) 

REM :: Predict size based on average sample bitrate minus (!) the error
REM @FOR /F "tokens=*" %%A IN (%tmp_enc_audio_and_rest%) DO @SET enc_audio_and_rest_size=%%~zA

REM @CALL :PREDICT_SIZE "%duration_sec%", "%autosize_predicted_bitrate%-%autosize_predicted_bitrate_error%", "", "%enc_audio_and_rest_size%", predicted_file_size
REM @ECHO Predicted file size: %predicted_file_size% bytes

REM @CALL :DETERMINE_MEDIUM "%predicted_file_size%", targetsize, targetsize_formated
REM @IF "%targetsize%" EQU "0" ECHO Auto target size set to 0KiB --^> exiting... & GOTO CLEAN_UP
REM @ECHO Auto target size set to %targetsize_formated%

REM :SKIP_AUTO_SIZE_CALCULATION 

:: Bitrate calculation is ONLY done if there is already targetsize specified!
@IF "%targetsize%" EQU "" GOTO SKIP_BITRATE_CALCULATION
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO Starting bitrate calculation
@ECHO ---------------------------------------------------------------------------
:: kan ook gebruikt worden: FOR /F "usebackq" %%A IN ('%tmp_enc_audio_and_rest%') DO set size=%%~zA
:: See here: http://stackoverflow.com/questions/1199645/how-can-i-check-the-size-of-a-file-in-a-windows-batch-script

:: calculate original video bitrate, needed to calculate the new bitrate:
@CALL :EVALUATE "%orig_video_size%*8/%duration_sec%/1000"
@SET orig_video_bitrate=%result%

:: calculate new video-only size
@CALL :EVALUATE "%targetsize%-%enc_audio_and_rest_size%"
@SET expected_video_size=%result%

:: ratio between old and new video-only sizes = ratio between new and old bitrate --> calculate new bitrate
@CALL :EVALUATE "%orig_video_bitrate%*%expected_video_size%/%orig_video_size%+0.5"
@SET new_bitrate=%result%

:: convert new bitrate to integer
@CALL :EVALUATE "clng(%new_bitrate%)"
@SET suffix=bSize%result%%suffix%

:: print info
@ECHO Original size: %input_size% bytes (%orig_video_size% + %orig_audio_size% + %orig_rest_size% bytes)
@ECHO Original bitrate: %orig_video_bitrate% kbit/s
@IF "%there_exist_subs%" EQU "true" (
	@ECHO New size: %targetsize% bytes (%expected_video_size% + %enc_audio_size% + %orig_rest_size% bytes)
) ELSE (
	@ECHO New size: %targetsize% bytes (%expected_video_size% + %enc_audio_and_rest_size% bytes)
)
@ECHO New bitrate: %new_bitrate% kbit/s
 
:SKIP_BITRATE_CALCULATION
 
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO Starting encoding process
@ECHO ---------------------------------------------------------------------------
 
@IF "%bitrate_control%" EQU "" @SET bitrate_control=--vb %new_bitrate% --two-pass
 
@CALL :HANDBRAKE %tmp_orig_video%, %tmp_enc_video%, "" , "%bitrate_control%", "%useropt%", "LOW" 3>&2 2>&1 1>&3
@ECHO Errorlevel: %ERRORLEVEL%
 
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO Making picture samples
@ECHO ---------------------------------------------------------------------------
 
@GOTO END_LOOP_PICTURE_SAMPLES
:: Make picture samples
@SET picture_sample_nr=0
 
:LOOP_PICTURE_SAMPLES
@SET /A picture_sample_nr=%picture_sample_nr%+1
 
@CALL :EVALUATE "clng(((%duration_sec%)/(%nr_picture_samples%*2))*(2*%picture_sample_nr%-1))"
@SET time_offset=%result%
@ECHO Making sample picure %picture_sample_nr%.
 
@START /B /LOW /AFFINITY %affinity% /WAIT ffmpeg %ffmpeg_options% -loglevel error -ss %time_offset% -i %tmp_orig_video% -frames 1 "%tmp_folder%\sample_%picture_sample_nr%_original.bmp" 
@IF %ERRORLEVEL% NEQ 0 ECHO Cannot make sample picture.
 
@START /B /LOW /AFFINITY %affinity% /WAIT ffmpeg %ffmpeg_options% -loglevel error -ss %time_offset% -i %tmp_enc_video% -frames 1 "%tmp_folder%\sample_%picture_sample_nr%_encoded.bmp" 
@IF %ERRORLEVEL% NEQ 0 ECHO Cannot make sample picture.
 
@IF %picture_sample_nr% LSS %nr_picture_samples% GOTO LOOP_PICTURE_SAMPLES
 
:END_LOOP_PICTURE_SAMPLES
@ECHO Finished making sample pictures.
 
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO Starting muxing process
@ECHO ---------------------------------------------------------------------------

:: Generate unique suffix for output in finish directory if the file allready exists.
:LOOP_FINAL_NAME
@IF NOT EXIST "%destination_folder%\%input_name%%mux_nr_suffix%.mkv" GOTO FINAL_NAME_UNIQUE
@IF NOT DEFINED mux_nr SET mux_nr=0
@SET /A mux_nr=%mux_nr%+1
@SET mux_nr_suffix=-0%mux_nr%
@GOTO LOOP_FINAL_NAME
:FINAL_NAME_UNIQUE
 
@mkvmerge --priority lower --ui-language English -o "%destination_folder%\%input_name%%mux_nr_suffix%.mkv" %tmp_enc_video% %tmp_enc_audio_and_rest% | %filter%
@ECHO Error: %ERRORLEVEL% 
@IF %ERRORLEVEL% NEQ 0 GOTO FINISH
 
@ECHO.
@mkvmerge --ui-language English --identify "%destination_folder%\%input_name%%mux_nr_suffix%.mkv" 
@IF %ERRORLEVEL% EQU 2 GOTO FINISH
 
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO Comparing...
@ECHO ---------------------------------------------------------------------------
 
@mkvinfo --ui-language English "%destination_folder%\%input_name%%mux_nr_suffix%.mkv" > %tmp_enc_mkvinfo%
@IF %ERRORLEVEL% NEQ 0 GOTO ERROR_MKVINFO
 
@fc %tmp_orig_mkvinfo% %tmp_enc_mkvinfo%
 
:CLEAN_UP
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO Cleaning up...
@ECHO ---------------------------------------------------------------------------
 
@IF EXIST "%tmp_folder%\*.mkv" @del "%tmp_folder%\*.mkv" & @ECHO Temporary mkv files removed
 
:FINISH
@ECHO.
@ECHO ---------------------------------------------------------------------------
@ECHO Finished
@ECHO Timestamp: %date% %time%
@ECHO ---------------------------------------------------------------------------
 
:END
@GOTO QUIT
 
:ERROR_NO_DURATION
@ECHO.
@ECHO ERROR: Duration can not be found.
@GOTO CLEAN_UP
 
:ERROR_MKVINFO
@ECHO.
@ECHO MKVINFO returned errorcode: %ERRORLEVEL%
@GOTO QUIT
 
:ERROR_NO_ARG
@ECHO.
@ECHO ERROR: No file selected to encode.
@GOTO QUIT
 
:ERROR_BAD_ARG
@ECHO.
@ECHO ERROR: File selected for encode can not be found.
@GOTO QUIT
 
:ERROR_TARGET_FOLDER_NOT_EXIST
@ECHO.
@ECHO ERROR: Target folder doesn't seem to exist. Exiting...
@GOTO QUIT
 
:MOVE_LOG
@IF EXIST %log_file% MOVE /Y %log_file% "%destination_folder%\%input_name%%mux_nr_suffix%.encode.log" >NUL
@GOTO EOF

::---------------------------------------------------------------------------------------------------------------------------------------------------------------------
::FUNCTIONS
::---------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
:HANDBRAKE			-- runs handbrake
::					-- %~1: input-path
::					-- %~2: output-path
:: 					-- %~3: start-stop-command
::					-- %~4: bitrate command
::					-- %~5: user command
::					-- %~6: priority
:: handbrake uses stderr (2) to show encode info, and stdout (1) to show progress. | only takes stdout (1), not (2). Because wtee doesn't like progress, we need to switch those two.
:: redirects are read from right to left! So first we redirect stdout (1) to 3 then stderr (2) to 1, then 3 to stderr (2). Stderr is not passed with pipe, only stdout.
 
@START /B /%~6 /AFFINITY %affinity% /WAIT HandBrakeCLI %~3 -i %1 -o %2 -e x264 --x264-profile high --h264-level 4.1 --x264-tune film --audio "none" %~5 %~4 --no-dvdnav %testoptions% 
@IF %ERRORLEVEL% NEQ 0 ECHO Handbrake exited with an error. & GOTO CLEAN_UP
@GOTO:EOF
 
:EVALUATE			-- evaluate with VBS and return to result variable
::					-- %~1: VBS string to evaluate
:: Extra info: http://groups.google.com/group/alt.msdos.batch.nt/browse_thread/thread/9092aad97cd0f917
:: Default  SetLocale() = 2067, d.i. voor belgie. VBS interpreteert dan een punt als een decimaal teken,
:: wat OK is, maar geeft het resultaat terug met een komma. Een getal met een komma kan VBS
:: blijkbar niet als een getal interpreteren, en geeft een error. SetLocale(1033) zorgt dat de
:: decimale teken een punt is, zodat het getal terug hergebruikt kan worden in VBS.
 
@IF [%1]==[] ECHO Input argument missing & GOTO :EOF 
::@ECHO "%~1"
 
@ECHO SetLocale(1033) : wsh.echo "result="^&eval("%~1") > %temp%\evaluate_tmp_67354.vbs 
@FOR /f "delims=" %%a IN ('cscript //nologo %temp%\evaluate_tmp_67354.vbs') do @SET "%%a" 
@DEL %temp%\evaluate_tmp_67354.vbs
::@ECHO EVAL: "%~1" RESULT: "%result%"
 
@GOTO:EOF
 
 
:CALCULATE_AVERAGE	-- calculates the average of bitrates stored in a file
::					-- %~1: path to file
::					-- %~2: OUT resulting bitrate
@SETLOCAL
:: Concatenate values to a string
@SET concat_bitrates=0
@FOR /F "usebackq tokens=2" %%L IN (%1) DO @SET concat_bitrates=!concat_bitrates!+%%L

:: Calculate the (average) bitrate
@CALL :EVALUATE "(%concat_bitrates%)/%nr_samples_for_autosize%"
@(ENDLOCAL 
	SET "%~2=%result%"
)
@GOTO:EOF
 
 
:MAKE_SAMPLES		-- function that makes samples
::					-- %~1: input
::					-- %~2: output_suffix
::					-- %~3: handbrake options
::					-- %~4: print info [Y/N], default: N - prints % done, Y gives old prints
::					-- %~5: OUT average bitrate
:: Make samples and put their bitrates into a file.
@IF EXIST %tmp_sample_bitrates_log% @del %tmp_sample_bitrates_log%
@SET sample_nr=0
 
:LOOP_ENCODE_SAMPLES
:: Print some info on progres
@CALL :EVALUATE "CInt(%sample_nr%/%nr_samples_for_autosize%*100)"
@IF "%~4" EQU "N" <nul set /p =Working... %result%%% done
 
@SET /A sample_nr=%sample_nr%+1
 
@CALL :EVALUATE "((%duration_sec%-%sample_duration_sec%)/(%nr_samples_for_autosize%*2))*(2*%sample_nr%-1)"
@SET start_time=%result%
@IF "%~4" EQU "Y" @ECHO Starttime of sample %sample_nr% is %start_time% sec.
 
@IF "%~4" EQU "Y" (@SET "print_redirect=1>&2") ELSE (@SET "print_redirect=1>nul")
@CALL :HANDBRAKE %1, "%tmp_folder%\%~2_%sample_nr%.mkv", "--start-at duration:%start_time% --stop-at duration:%sample_duration_sec%", "--quality %auto_size_rf%", "%~3" %print_redirect%, "LOW" 2>%tmp_sample_activity_log%
@FOR /F "tokens=8 delims=:" %%T IN ('FIND "kb/s:" ^< %tmp_sample_activity_log%') DO @(
	@IF "%~4" EQU "Y" (@ECHO Bitrate: %%T kbit/s) 
	@ECHO Sample%sample_nr%: %%T kbit/s >> %tmp_sample_bitrates_log%
)

:: We kunnen bitrates ook anders berekenen: door gewoon te kijken naar de video-filesize (!) zoals reeds gedaan boven:
:: @FOR /F "tokens=*" %%A IN ("%tmp_folder%\tmp_video_sample%sample_nr%.mkv") DO @SET sample_size=%%~zA
:: @CALL :EVALUATE "%sample_size%*8/20/1000"
:: @ECHO Bitrate based on size: %result% kbit/s
 
@IF "%~4" EQU "Y" @ECHO.
@IF "%~4" EQU "N" <nul set /p "=%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%%bs_char%"
 
@IF %sample_nr% LSS %nr_samples_for_autosize% GOTO LOOP_ENCODE_SAMPLES
@IF "%~4" EQU "Y" @ECHO Finished making samples.
 
@CALL :CALCULATE_AVERAGE %tmp_sample_bitrates_log%, %~5
@GOTO:EOF
 
 
:DETERMINE_MEDIUM	-- this function takes a size and returns the smallest possible medium size
::					-- %~1: source size in bytes
::					-- %~2: OUT medium size that fits the source
::					-- %~3: OUT user friendly medium size that fits the source
:: Choose as size any size which is greater than the predicted, and start encoding!
 
@IF "%~1" EQU "" @(
	@ECHO No source size specified to check against medium.
	@SET "%~2=0"
	@SET "%~3=0KiB"
	@GOTO:EOF
)	
 
@CALL :EVALUATE "%~1<=%CD-28P80min%"
@IF %result% EQU True SET "%~2=%CD-28P80min%" & SET "%~3=25MiB" & @GOTO :EOF
 
@CALL :EVALUATE "%~1<=%CD-14P80min%"
@IF %result% EQU True SET "%~2=%CD-14P80min%" & SET "%~3=50MiB" & @GOTO :EOF
 
@CALL :EVALUATE "%~1<=%CD-7P80min%"
@IF %result% EQU True SET "%~2=%CD-7P80min%" & SET "%~3=100MiB" & @GOTO :EOF
 
@CALL :EVALUATE "%~1<=%CD-2H80min%"
@IF %result% EQU True SET "%~2=%CD-2H80min%" & SET "%~3=175MiB" & @GOTO :EOF
 
@CALL :EVALUATE "%~1<=%CD-H80min%"
@IF %result% EQU True SET "%~2=%CD-H80min%" & SET "%~3=350MiB" & GOTO :EOF
 
@CALL :EVALUATE "%~1<=%CD-80min%"
@IF %result% EQU True SET "%~2=%CD-80min%" & SET "%~3=700MiB" & GOTO :EOF
 
@CALL :EVALUATE "%~1<=%CD-80minX2%"
@IF %result% EQU True SET "%~2=%CD-80minX2%" & SET "%~3=1.37GiB" & GOTO :EOF
 
@CALL :EVALUATE "%~1<=%CD-80minX3%"
@IF %result% EQU True SET "%~2=%CD-80minX3%" & SET "%~3=2.05GiB" & GOTO :EOF
 
@CALL :EVALUATE "%~1<=%DVD-SL%"
@IF %result% EQU True SET "%~2=%DVD-SL%" & SET "%~3=4.37GiB" & GOTO :EOF
 
@CALL :EVALUATE "%~1<=%DVD-DL%"
@IF %result% EQU True SET "%~2=%DVD-DL%" & SET "%~3=7.95GiB" & GOTO :EOF
 
@CALL :EVALUATE "%~1<=%Bluray-HSL%"
@IF %result% EQU True SET "%~2=%Bluray-HSL%" & SET "%~3=12.5GB" & GOTO :EOF
 
@CALL :EVALUATE "%~1<=%Bluray-TDL%"
@IF %result% EQU True SET "%~2=%Bluray-TDL%" & SET "%~3=15.52GB" & GOTO :EOF
 
@CALL :EVALUATE "%~1<=%Bluray-SL%"
@IF %result% EQU True SET "%~2=%Bluray-SL%" & SET "%~3=25GB" & GOTO :EOF
 
@CALL :EVALUATE "%~1<=%Bluray-DL%"
@IF %result% EQU True SET "%~2=%Bluray-DL%" & SET "%~3=50GB" & GOTO :EOF
 
@ECHO Auto Size could not be set. The filesize (%~1 bytes) is too large for any medium.
@SET "%~2=0" 
@SET "%~3=0KiB"
@GOTO:EOF
 
:PREDICT_SIZE		-- predicts the file size based on bitrates 
::					-- %~1: duration in seconds
::					-- %~2: bitrate (kbit/s) of video
::					-- %~3: bitrate (kbit/s) of audio
::					-- %~4: size (bytes) of rest
::					-- %~5: OUT (bytes) resulting size
@SETLOCAL
@SET d_sec=%~1
@SET v_bitrate=%~2
@SET a_bitrate=%~3
@SET r_bytes=%~4

:: Check input
@IF "%d_sec%" EQU "" @ECHO ERROR: duration = 0sec
@IF "%v_bitrate%" EQU "" @SET v_bitrate=0
@IF "%a_bitrate%" EQU "" @SET a_bitrate=0
@IF "%r_bytes%" EQU "" @SET r_bytes=0

:: Calculate
@CALL :EVALUATE "((%v_bitrate%)+(%a_bitrate%))*(%d_sec%)*1000/8+(%r_bytes%)"
 
@(ENDLOCAL 
	SET "%~5=%result%"
)
@GOTO :EOF
 
:TRIM_SPACES retval string -- trims spaces around string and assigns result to variable
::                         -- retvar [out] variable name to store the result in, no (!) quotes
::                         -- string [in]  string to trim, no (!) quotes
:: source http://www.dostips.com
@FOR /f "tokens=1*" %%A in ("%*") do @set "%%A=%%B"
@GOTO :EOF
 
:SEEK_IN_MAP key !map! out_var	-- returns value from key in map
::					-- %~1: variable key name.
::					-- %~2: map variable, possibly with delayed expansion.
::					-- %~3: OUT variable name in which value will be returned.
@SETLOCAL
@ECHO OFF
SET key=%~1
SET map=%~2

:: Extract value
CALL SET temp_val=%%map:*%key%--=%%
SET value=%!!&%

:: Validity check
IF "%value:--=%" NEQ "%value%" @( 
ECHO Wrong value ^("%value%"^) with key ^("%key%"^) returned from map ^("%map%"^)
GOTO QUIT )
 
@( 	ENDLOCAL
	SET "%~3=%value%"
)
@GOTO :EOF
 
:MAKE_AUDIO_MAP map path
::					-- %~1: OUT map
::					-- %~2: path of the FFMPEG stream txt
@SETLOCAL
@ECHO OFF
SET map=
SET file=%2
SET nr=0
 
FOR /F "tokens=4-9 delims=,:" %%T IN ('TYPE %file% ^| FINDSTR /I /R "^.*Stream.*Audio.*$"') DO (
	REM @ECHO tokens: %%T, %%U, %%V, %%W, %%X
	REM codec
	CALL :TRIM_SPACES tmp_var %%T
	SET "map=codec!nr!.0--!tmp_var!;!map!"
	REM frequency has HZ at the end and should be removed.
	CALL :TRIM_SPACES tmp_var %%U
	FOR /F "tokens=1 delims= " %%F IN ("%%U") DO ( CALL :TRIM_SPACES tmp_var %%F )
	SET "map=frequency!nr!.0--!tmp_var!;!map!"
	REM channels
	CALL :TRIM_SPACES tmp_var %%V
	SET "map=channels!nr!.0--!tmp_var!;!map!"
	REM sample resolution
	CALL :TRIM_SPACES tmp_var %%W
	SET "map=sampleres!nr!.0--!tmp_var!;!map!"
	REM bitrate has kbit/s at the end, and possibly default
	IF "%%X" EQU "" (
		SET "map=bitrate!nr!.0--64;!map!"
		ECHO WARNING^^! Bitrate of track !nr! is unknown and is set to 64 kbit/s.
	) ELSE (
		FOR /F "tokens=*" %%C IN ('ECHO %%X ^| FIND /I /C "kb/s"') DO (
			IF "%%C" EQU "1" ( REM dus er is effectief sprake van een xxxkb/s of dergelijk
				FOR /F "tokens=1 delims= " %%B IN ("%%X") DO ( CALL :TRIM_SPACES tmp_var %%B )
				SET "map=bitrate!nr!.0--!tmp_var!;!map!"
			)
		)
		REM default selection
		FOR /F "tokens=*" %%C IN ('ECHO %%X ^| FIND /I /C "default"') DO ( IF "%%C" EQU "1" SET "map=default--!nr!.0;!map!" )
	)
	SET /A nr=!nr!+1
)
@( 	ENDLOCAL
	SET "%~1=%map%streams--%nr%"
)
@GOTO :EOF
 
:AUDIO_MAP_UPDATE	-- checks the converson string and updates the audio map
::					-- %~1: conversion_string
::					-- %~2: track number
::					-- %~3: OUT conv_nr variable name with last used conversion number
::					-- %~4: OUT audio map name
@SETLOCAL
@FOR /F "tokens=1-2 delims=:" %%X IN ("%~1") DO (
@SET "check=%%X"
@SET "conversion=%%Y"
)
:: @ECHO "CONVERSION: %conversion%"
 
@CALL :EVALUATE "%check%" & REM If this is true, it means that this track can be encoded with the codec.
:: @ECHO "CHECK: %check% --> %result%"
 
@IF "%result%" EQU "True" (
	SET /A conv_track_nr=!%~3!+1
	SET orig_track_nr=%~2
	SET "conversion=%conversion:--=!orig_track_nr!.!conv_track_nr!--%" 
)
 
(@ENDLOCAL
	IF "%result%" EQU "True" SET "%~4=!%~4!;%conversion%" & SET "%~3=%conv_track_nr%"
)	
@GOTO :EOF
 
 
:SCRAMBLE track, coordinate 		-- finds all the comination between the original and compressible tracks
::					-- %~1: track number: one should begin with 0
::					-- %~2: coordinate-part, which is passed by by the scramble interation
:: One should call the scramble function without arguments
@SETLOCAL
@SET track_nr=%~1
@IF "%track_nr%" EQU "" SET track_nr=0
@CALL :SEEK_IN_MAP conversions%track_nr%, "%audio_map%", nr_conv
@IF "%track_nr%" GTR "%nr_max_stream%" ECHO Scramble track too high & GOTO :EOF
 
@FOR /L %%C IN (0,1,%nr_conv%) DO (
	IF "%track_nr%" EQU "%nr_max_stream%" ( REM This means the coordinates should be written. 
		SET "scram=!scram!;%~2,%track_nr%.%%C"
	) ELSE (
		SET /A next_track=%track_nr%+1
		IF "%~2" EQU "" ( 
			CALL :SCRAMBLE !next_track!, "%track_nr%.%%C"
		) ELSE (
			CALL :SCRAMBLE !next_track!, "%~2,%track_nr%.%%C"
		)
	)
)
(@ENDLOCAL
	SET "scramble=%scramble%%scram%"
	IF "%track_nr%" EQU "0" SET scramble=!scramble:~1!
)	
@GOTO :EOF
 
:PRINT_SCRAMBLE video_bitrate		-- prints file sizes based on the video bitrate and audio scramble
::					-- %~1: video bitrate
::					-- %~2: prefix
@SETLOCAL
@SET "video_bitrate=%~1"
@CALL :EVALUATE "FormatNumber(%video_bitrate%)"
@ECHO %~2Average video bitrate: %result% kbit/s
@FOR /F "tokens=1" %%i in ("%!!!!!!%") do @( REM For each conversion possibility the loop begins here
	@SET "audio_str= | + "
	@SET "bitrates="
	@FOR %%j in (%%i) do @(
		@CALL :SEEK_IN_MAP codec%%j, "%audio_map%", codec
		@CALL :SEEK_IN_MAP bitrate%%j, "%audio_map%", bitrate
		@SET "audio_str=!audio_str![!codec! ^@!bitrate!kbit/s], "
		@SET "bitrates=!bitrates!!bitrate!+"
	)
	@SET "audio_str=!audio_str:~0,-2!"
	@SET "bitrates=!bitrates:~0,-1!"
	@CALL :PREDICT_SIZE "%duration_sec%", "%video_bitrate%", "!bitrates!", "", size
	@CALL :DETERMINE_MEDIUM !size!, medium, medium_formated
		REM extra calculatesize for minimum bitrate
		@CALL :EVALUATE "%video_bitrate% - (%video_bitrate%*%percentage_autosize_bitrate_error%/100)"
		@CALL :PREDICT_SIZE "%duration_sec%", "!result!", "!bitrates!", "", min_size
		@CALL :DETERMINE_MEDIUM !min_size!, min_medium, min_medium_formated
	@ECHO %~2!audio_str! --^> !medium_formated! ^(!min_medium_formated!^)& REM Om een of andere rede print deze laatste haak niet zonder escape
)
@ENDLOCAL
@GOTO :EOF
 
:NEGLECT_TRACK_CMDLINE		-- generates command line for neglected tracks
::					-- %~1: "Audio", "Subtitle" or "Video": based on ffmpeg stream naming scheme
::					-- %~2: comma delimited user input of tracks to neglect
::					-- %~3: OUT variable name into which to put the ffmpeg command line part for removing selected tracks
::					-- %~4: OUT variable name for error code: 0 = OK, else not OK
:: This function iterates through all selected tracks and returns the command line for fmpeg to ignore it during extraction
:: It also checks whether the track exists. If not, it returns an error.
@SETLOCAL
@SET err=0
@FOR %%N IN (%~2) DO @(
	REM Sometimes there is no stream name like "(nld)", so streamnumber looks like "#0:3: Audio:" instead of "#0:2(nld): Audio:".
	REM That is wht we use "[:(]" after stream number.
	@FOR /F %%C IN ('TYPE %tmp_orig_identify_ffmpeg% ^| FINDSTR /I /R "#0:%%N[:(].*%~1:" ^| FIND /V /C ""') DO @(
		@IF "%%C" EQU "0" @( @ECHO %~1 track ID %%N does not exist. & @SET err=1 )
		@IF "%%C" GTR "1" @( @ECHO %~1 track ID %%N was found more than once. & @SET err=1 )
	)
	@MOVE %tmp_pre-enc_identify_ffmpeg% %tmp_identify% > NUL
	@TYPE %tmp_identify% | @FINDSTR /I /V /R "#0:%%N.*%~1:" > %tmp_pre-enc_identify_ffmpeg%
	@DEL %tmp_identify% 
	@SET cmd_line=!cmd_line! -map -0:%%N
)
@(ENDLOCAL
	SET %~3=%cmd_line%
	SET %~4=%err%
)
@GOTO :EOF
 
:PRINT_ANALYSIS		-- function used for compression analysis printing
::					-- %~1: resolution_suffix
::					-- %~2: cmd-line options
@SETLOCAL
@SET "res_height_suffix=%~1"
@SET "cmd_line_options=%~2"
 
@ECHO - Denoise: off
@CALL :MAKE_SAMPLES %input%, "tmp_video_sample_%res_height_suffix%", "%useropt% %cmd_line_options%", "N", avg_bitrate
@CALL :PRINT_SCRAMBLE "%avg_bitrate%", "   "
 
@IF "%choice_decomb_analysis%" EQU "2" @(
	@ECHO.
	@ECHO - Denoise: off - Decomb: on
	@CALL :MAKE_SAMPLES %input%, "tmp_video_sample_%res_height_suffix%_decomb", "%useropt% %cmd_line_options% --decomb", "N", avg_bitrate_decomb
	@CALL :PRINT_SCRAMBLE "!avg_bitrate_decomb!", "   "
)
 
@IF "%choice_denoise_analysis%" EQU "2" @(
	@ECHO.
	@ECHO - Denoise: weak (predicted^)
	@CALL :MAKE_SAMPLES %input%, "tmp_video_sample_%res_height_suffix%_denoise-s", "%useropt% %cmd_line_options% --denoise=strong", "N", avg_bitrate_ds
	@CALL :EVALUATE "(0.65954)*(!avg_bitrate!-!avg_bitrate_ds!) + !avg_bitrate_ds!"
	@CALL :PRINT_SCRAMBLE "!result!", "   "
	@IF "%choice_decomb_analysis%" EQU "2" @(
		@ECHO.
		@ECHO - Denoise: weak (predicted^) - Decomb: on
		@CALL :MAKE_SAMPLES %input%, "tmp_video_sample_%res_height_suffix%_denoise-s_decomb", "%useropt% %cmd_line_options% --denoise=strong --decomb", "N", avg_bitrate_ds_decomb
		@CALL :EVALUATE "(0.65954)*(!avg_bitrate_decomb!-!avg_bitrate_ds_decomb!) + !avg_bitrate_ds_decomb!"
		@CALL :PRINT_SCRAMBLE "!result!", "   "
	)
	@ECHO.
	@ECHO - Denoise: medium (predicted^)
	@CALL :EVALUATE "(0.5138)*(!avg_bitrate!-!avg_bitrate_ds!) + !avg_bitrate_ds!"
	@CALL :PRINT_SCRAMBLE "!result!", "   "
	@IF "%choice_decomb_analysis%" EQU "2" @(
		@ECHO.
		@ECHO - Denoise: medium (predicted^) - Decomb: on
		@CALL :EVALUATE "(0.5138)*(!avg_bitrate_decomb!-!avg_bitrate_ds_decomb!) + !avg_bitrate_ds_decomb!"
		@CALL :PRINT_SCRAMBLE "!result!", "   "
	)
	@ECHO.
	@ECHO - Denoise: strong
	@CALL :PRINT_SCRAMBLE "!avg_bitrate_ds!", "   "
	@IF "%choice_decomb_analysis%" EQU "2" @(
		@ECHO.
		@ECHO - Denoise: strong - Decomb: on
		@CALL :PRINT_SCRAMBLE "!avg_bitrate_ds_decomb!", "   "
	)
)
 
@IF "%choice_denoise_analysis%" EQU "3" @(
	@ECHO.
	@ECHO - Denoise: weak
	@CALL :MAKE_SAMPLES %input%, "tmp_video_sample_%res_height_suffix%_denoise-w", "%useropt% %cmd_line_options% --denoise=weak", "N", avg_bitrate
	@CALL :PRINT_SCRAMBLE "!avg_bitrate!", "   "
	@IF "%choice_decomb_analysis%" EQU "2" @(
		@ECHO.
		@ECHO - Denoise: weak - Decomb: on
		@CALL :MAKE_SAMPLES %input%, "tmp_video_sample_%res_height_suffix%_denoise-w_decomb", "%useropt% %cmd_line_options% --denoise=weak --decomb", "N", avg_bitrate_decomb
		@CALL :PRINT_SCRAMBLE "!avg_bitrate_decomb!", "   "
	)
	@ECHO.
	@ECHO - Denoise: medium
	@CALL :MAKE_SAMPLES %input%, "tmp_video_sample_%res_height_suffix%_denoise-m", "%useropt% %cmd_line_options% --denoise=medium", "N", avg_bitrate
	@CALL :PRINT_SCRAMBLE "!avg_bitrate!", "   "
	@IF "%choice_decomb_analysis%" EQU "2" @(
		@ECHO.
		@ECHO - Denoise: weak - Decomb: on
		@CALL :MAKE_SAMPLES %input%, "tmp_video_sample_%res_height_suffix%_denoise-m_decomb", "%useropt% %cmd_line_options% --denoise=medium --decomb", "N", avg_bitrate_decomb
		@CALL :PRINT_SCRAMBLE "!avg_bitrate_decomb!", "   "
	)
	@ECHO.
	@ECHO - Denoise: strong
	@CALL :MAKE_SAMPLES %input%, "tmp_video_sample_%res_height_suffix%_denoise-s", "%useropt% %cmd_line_options% --denoise=strong", "N", avg_bitrate
	@CALL :PRINT_SCRAMBLE "!avg_bitrate!", "   "
	@IF "%choice_decomb_analysis%" EQU "2" @(
		@ECHO.
		@ECHO - Denoise: strong - Decomb: on
		@CALL :MAKE_SAMPLES %input%, "tmp_video_sample_%res_height_suffix%_denoise-s_decomb", "%useropt% %cmd_line_options% --denoise=strong --decomb", "N", avg_bitrate_decomb
		@CALL :PRINT_SCRAMBLE "!avg_bitrate_decomb!", "   "
	)
)
 
(@ENDLOCAL & REM -- RETURN VALUES
    REM IF "%~1" NEQ "" SET %~1=%LocalVar1%
    REM IF "%~2" NEQ "" SET %~2=%LocalVar2%
)
@GOTO:EOF
 
:myFunctionName		-- proto-function
::					-- %~1: argument description here
@SETLOCAL
:: --function body here
set LocalVar1=...
set LocalVar2=...
(@ENDLOCAL & REM -- RETURN VALUES
    IF "%~1" NEQ "" SET %~1=%LocalVar1%
    IF "%~2" NEQ "" SET %~2=%LocalVar2%
)
@GOTO:EOF

::---------------------------------------------------------------------------------------------------------------------------------------------------------------------
::END FUNCTIONS
::---------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
 
:QUIT
::@ENDLOCAL
@PAUSE
@ECHO.
:EOF

15/07/2012

Deus Ex: Human Revolution – some thoughts from a fan

Filed under: Gaming — Tags: , — Davor @ 20:36

I played the original twice. When I was 17, and later when I was about 22. I am not really into shooters – yes, I skipped the Invisible War – but this game is so much more. One gets immersed into the future as seen by its creator Warren Spector. To describe this “not the end of the world but you can see it from there”-feeling would take many pages. So I’ll keep it short. Beside the great gameplay, sound and story, Deus Ex confronts you with moral dilemmas and makes you think about human augmentation and just polity. Beside the entertainment value it also has some “educational” value.

So now we have a new prequel Deus Ex: HR. Usually I play a game 2-4 years after its release, when it’s close to bugfree and possibly improved by the mods. But after I saw the Deus Ex: HR trailer and the intro I just couldn’t resist… So here is what I liked and did not like about the game. It might contain spoilers.

The visuals are stunning and memorable. Some are simply jawdropping. The creativity involved in making the world look mystical, but at the same time futuristic and “Deus Ex” is really top notch. These artists definitely understand the original.

Just like the original, this Deus Ex has definitely a memorable soundtrack. Very futuristic and at the same time “Deus Ex”. The voice acting is also incredibly well done, for all the NPC, but especially the main character and main NPC’s.

Overall gameplay is good to very good/perfect. Although this game seems to be made for console, it plays very well on PC too. It simply feels right after you get used to the gameplay. It is also obvious that much time has been spent on making the gameplay flawless, and incorporating various ways for one to get to the objective.

The dialogues are much improved compared to the original. There is much more interaction and choice left to the player. The social interaction aug is also very well implemented. The thing that bothered me were the sometimes shallow facial expressions.

Story is OK for a Deus Ex game. Whereas the first Deus Ex dealt with the just governmental system, the subject of HR is more about the dangers of augmentation and human transcendence. From that perspective it seems that the end of HR was pushed to resemble the original Deus Ex ending where one could choose – and play God. I did miss some depth and consistency. The story seems artificially prolonged because the plot is too simple, and at times becomes boring. So although the plot of HR is good and has great potential, the presentation of it seems far less to me. I think I missed the message during my 60h play. The original did better here, although it didn’t had as much content – I think.

Quake-like boss fights? Well, I don’t think they have a place in a Deus Ex game. Also it is a pity that these bosses are completely void of personality – in contrast with the other NPC.

The skill system is removed, and now completely replaced with the augmentation system. I really liked the skill system: it gave one a certain level of specialization that is unrelated to augs. Like better aiming with weapons you specialize in, faster movement with heavy weapons, staying longer under water, some hacking skills, medical skills, etc. So, to me this was definitely a big loss. An unique playing experience could be highly increased with a skill system.

The augs now get unlocked with XP-points. Most are well designed and well implemented in the game. The fact that you already have them all removes the “wauw” effect of finding one though. Auto-recharging one battery is a good improvement from the previous series – now one can experiment without permanently using up the bio-energy. But recharging only one seems inconsistent: having two charged cells and taking one enemy down will “waste” this second cell, while doing the same with one cell will not – the empty cell will reload eventually.

The praxis packs (and other items) scattered through the game on sometimes strange locations… it could have been better. Same with the empty lockers…

I also missed a way to replay (or re-read) the conversations I had with people. Sometimes there is much information at once, and you don’t want to miss a thing… I though the original had a log for this.

The hacking is improved – resembles System Shock (?) – but is imbalanced in my opinion. With a few aug-points one can hack everything and get money and XP for free. One can misuse this eventually. On hardest settings I had eventually more than 40 stops! and viruses in my backpack, and gathered very much gold. Even the toughest terminals went down easily. Such hacking makes searching for passcodes completely irrelevant.

Hacked turrets and robots have no cameras. They could have them.

The stealth difficulty could be higher. The patrol routes could be extended, randomized, etc. Now the guards usually don’t go just far enough to see you. Also, the AI during alarm or hostile faze could be better. The guards don’t go too far in their search, and if they do, they sometimes go alone, which makes them easy to take down unnoticed. Guards also don’t notice when some of them are missing for a long time.

The third person view unlocked with the right mouse button is interesting but lacks explanation – certainly for a game that is trying to look as realistic as possible. How come Adam is able to have such a “field of view”?

At the same time there are much eBooks scattered through the world that refer to real-life happenings. It’s kind of strange… it doesn’t feel right.

The DLC – Missing Link – doesn’t run from the main game. This is a pity. The pre-order special text in the weapon descriptions I also found very bothering.

Replay value? Not so if you ask me. But hard to tell why though. Maybe because the game tries to be so realistic your every decision feels like it is definite. The game feels more like an onetime experience… which sticks!

So does it live up to the name? I think it does. To be honest: there are flaws. But the vocals and visuals are stunning. The experience is unique. The story is OK. The gameplay is quite ambitious, and speaking of value: it is definitely worth the money. I am happy that de developers also tried successfully new things and didn’t simply brought us the original in a new package – like with HL2 (!!). I think they should have gone even further – although this is always a big risk. Because of this, I find it not as “redefining” as the original was, but at the same time I cannot look past its aspirations. So, although there are definitely things that could have been better, these shortcomings don’t measure up against the love, care and attention for detail that have been put into this game.

Older Posts »

Powered by WordPress