matoski.com on matoski.com

Automatic collection of prometheus metrics, cadvisor and grafana in Docker Swarm

Tue, 27 Nov 2018 13:14:21 +0000

Last time in were exposing some metrics for a golang application, but exposing the data is useless without having a way to visualise it or even store it somewhere in a location so we can analyze it when needed.

Some of the tools that we can use to do this are

At this point, I will assume have a docker swarm cluster available and ready to use.

Let’s create a stack file that will allow us to replicate this one on another swarm easily. All the required resources are on github.com/ilijamt/gomeetupamsnov2018

We need some prerequisites first

Network for the monitoring applications
Node metrics exporter/prometheus
Prometheus
Grafana
cAdvisor
Application that will expose metrics

Let’s create a very simple application that will build a small service that will expose various metrics.

main.go

package main

import (
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
	"net/http"
)

func main() {
	prometheus.MustRegister(prometheus.NewGauge(prometheus.GaugeOpts{
		Name: "version",
		Help: "Version information about this service",
		ConstLabels: map[string]string{
			"version": "v1.1.51",
			"service": "demo",
		},
	}))

	http.Handle("/metrics", promhttp.Handler())
	if err := http.ListenAndServe(":2112", nil); err != nil {
		panic(err)
	}
}

We need to create 2 services off the code and build create the docker images, running these commands will build and tag the images

docker build demo-v1 -t gomeetup-demo-v1:latest
docker build demo-v2 -t gomeetup-demo-v2:latest

I think everyone should be able to understand the stack file bellow.

stack.yaml

version: "3"

networks:
  monitoring:
  api:

services:
  demo_service_v1:
    image: gomeetup-demo-v1:latest
    deploy:
      mode: replicated
      replicas: 6
    networks:
      - monitoring
      - api
  demo_service_v2:
    image: gomeetup-demo-v2:latest
    deploy:
      mode: replicated
      replicas: 4
    networks:
      - monitoring
      - api
  cadvisor:
    image: google/cadvisor:latest
    deploy:
      mode: global
    ports:
      - "8080:8080"
    volumes:
      - /var/lib/docker/:/var/lib/docker
      - /dev/disk/:/dev/disk
      - /sys:/sys
      - /var/run:/var/run
      - /:/rootfs
      - /dev/zfs:/dev/zfs
    networks:
      - monitoring
  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    volumes:
      - ./data/grafana:/var/lib/grafana:rw
    deploy:
      mode: replicated
      replicas: 1
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    networks:
      - monitoring
  prometheus:
    image: prom/prometheus:latest
    ports:
      - '9090:9090'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./data/prometheus:/prometheus:rw
    deploy:
      mode: replicated
      replicas: 1
    networks:
      - monitoring

We also need to create the configuration for prometheus so we can get the data automatically off the services.

global:
  evaluation_interval: 15s
  scrape_interval: 15s

scrape_configs:
  - job_name: services
    metrics_path: "/metrics"
    scrape_interval: 10s
    dns_sd_configs:
      - names:
          - 'tasks.demo_service_v1'
          - 'tasks.demo_service_v2'
        type: 'A'
        port: 2112
  - job_name: cadvisor
    metrics_path: /metrics
    scrape_interval: 30s
    dns_sd_configs:
      - names:
          - "tasks.cadvisor"
        type: 'A'
        port: 8080

What this will do is create 2 jobs that will collect the metrics, at their specified periods, the tasks. is something you can use in docker swarm to get all the IP addresses associated with the service so the node exporter/prometheus can get all the data without us having to manually configure it.

Now to deploy this stack we can just issue the command bellow.

docker stack deploy gomeetupamsnov -c stack.yml

This will spin up all the necessary services required for this to work, after which we will be able to access.

cAdvisor on port 8080
Prometheus on port 9090
Grafana on port 3000

Load up some dashboards, there are plenty available on grafana’s page.

And voila we have some dashboards with information relevant to us.

Golang process data regarding cAdvisor

Golang process data regarding the demo services

Docker data from cAdvisor

Next steps

Automatic configuration of jobs, we can use consul for this, so anytime a new service is added we pick it up right away
Adding node exporter instead of full prometheus inside the swarm (done for simplicity)
Setting up alerting based on our parameters

Golang remote profiling and flamegraphs

Tue, 02 Oct 2018 16:21:31 +0000

Quite often we are found with a challenge to troubleshoot something in production, or to see why is our application slow, or why isn’t it serving requests fast enough.

We can use golang tool pprof to troubleshoot our system.

Golang has a package called pprof that provides an HTTP server with runtime profiling data in the format expected by the visualisation tool. We need to make sure that this endpoint is never exposed as it can leak sensitive data.

Profiling most commonly allows us to optimize our application, by providing dynamic analysis of various aspects of the running application.

In Golang there are several profilers that we can use as of version 1.11, and they are

allocs: A sampling of all past memory allocations
block: Stack traces that led to blocking on synchronization primitives
cmdline: The command line invocation of the current program
goroutine: Stack traces of all current goroutines
heap: A sampling of memory allocations of live objects. You can specify the gc GET parameter to run GC before taking the heap sample.
mutex: Stack traces of holders of contended mutexes
profile: CPU profile. You can specify the duration in the seconds GET parameter. After you get the profile file, use the go tool pprof command to investigate the profile.
threadcreate: Stack traces that led to the creation of new OS threads
trace: A trace of execution of the current program. You can specify the duration in the seconds GET parameter. After you get the trace file, use the go tool trace command to investigate the trace.

Profile and optimize only if you know that you have a performance problem. Because premature optimization is not only a waste of your time, but it will also slow you down in the future if you have to refactor your brittle and finely-tuned code. Attaching the network profiler has negligible performance overhead, so we can run it in production with live traffic without any noticeable performance penalties, and it gives you insight into how your application works with live data.

Profiling

Let’s look at some profiles to get us started. Before we begin let’s clone the GitHub repository locally so we can check it out and run it.

$ mkdir $GOPATH/src/github.com/ilijamt -p
$ cd $GOPATH/src/github.com/ilijamt
$ git clone https://github.com/ilijamt/gomeetupamsoct2018
$ cd gomeetupamsoct2018

CPU Profile

The CPU profiler runs for 30s, and uses sampling to determine which functions spend the most of the CPU time. The go runtime stops the execution every 10ms and records the stack of all running goroutines. If you want to increase the sampling rate you can take a look at SetCPUProfileRate.

$ cd pprof
$ go build

Just running the application won’t really help us, as we need to generate some load on the application so the profiler can gather the CPU samples while it’s running under load.

First we need to run the compiled application, and then generate some load, while the ab commands are running we need to run go tool pprof pprof http://127.0.0.1:6060/debug/pprof/profile

$ ./pprof &
$ ab -k -c 8 -n 100000 "http://127.0.0.1:6060/concat/?str=test&count=50" &
$ ab -k -c 8 -n 100000 "http://127.0.0.1:6060/fib/?n=50&type=recursive" &
$ ab -k -c 8 -n 100000 "http://127.0.0.1:6060/fib/?n=50&type=iterative" &

So in a different terminal we need to run at the same time

$ go tool pprof pprof http://127.0.0.1:6060/debug/pprof/profile

After it finishes dumping you can examine it when you need using the command

$ go tool pprof -http=:8080 dumps/pprof.pprof.samples.cpu.001.pb.gz

Let’s examine the CPU profile and see what is going on, if you want to play around with it you can run the command above and check the interactive graph and play with it.

On the graph we can see that most of the CPU time was spent in fibRecursive function which calculates the fibonacci number recursively. Let’s take a look at the function implementation, to see what it’s doing.

func fibRecursive(n int64) int64 {
	if n < 2 {
		return n
	}
	return fibRecursive(n-1) + fibRecursive(n-2)
}

Now let’s take a look at the flame graph

We can also see from the flame graph that most of the time it’s the fibRecursive function that is slowing the application down.

We can drill even more down to see which part of fibIterative and fibRecursive is taking the most time.

As a comparison I’ve put both fibIterative and fibRecursive so we can see the difference in implementation.

We can see that fibRecursive takes on average 13.91 sec to run, and total 1.2 minutes, versus the fibIterative which is 20ms.

We can play with the pprof ui tool and investigate more in details about what is going on, but I leave that up to you the reader to do so.

Heap profile

$ ./pprof &
ab -k -c 80 -n 1000000 "http://127.0.0.1:6060/concat/?str=test&count=5000"

So in a different terminal we need to run at the same time, this is usually quite quick to create a heap profile unlike the CPU profile

$ go tool pprof pprof http://127.0.0.1:6060/debug/pprof/heap

After it finishes dumping you can examine it when you need using the command

$ go tool pprof -http=:8080 dumps/pprof.pprof.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz

After it finishes dumping we can investigate the data we have collected and check out the issues that we have in our current system

Let’s take a look at the data, there are four parameters that we can pass to check out the application details and in short the profiler too can show you either allocation counts or in memory use. If you are concerned with the amount of memory being used, you want to take a look at the in memory use metrics, but if you are want to know about the garbage collection you look at the allocations. For more details you can take a look at the memory profiler, the code has quite a lot of useful comments about how it works.

Let’s look at some of the data that we’ve collected

alloc_object, alloc_space

From the picture we can see that we have allocated quite a lot of object in contact, up to 61.7% of the objects allocated came from that function.

From the picture we can see that we have allocated a whole of objects up to a total of 340GB

inuse_object, inuse_space

But when we look at the inuse objects of the application we can see that concat is not the one that created the most objects, but bufio.NewReaderSize

But if we take a look at the inuse space, we can see that the concat function uses whole 19.23MB of memory which accounts for a total of 82.75% of the whole memory usage.

And if we drill down we can see where is the memory being allocated, and it’s on the line 17 on the picture, we can easily improve this one by creating a buffer and writing to it instead of prepending a new string each time, which creates new objects all the time. Just replacing the function concat with

func concatV2(str string, count int) (ret string) {
	data := bytes.NewBuffer([]byte{})
	for i := 0; i < count; i ++ {
        data.WriteString(str)
	}
	return data.String()
}

We can verify that we are doing less allocations now and we are actually using less memory, I leave it up to you to check out the details, there is already a dump in dumps/pprof.pprof.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz in the GitHub repo that you can use right away.

Go routines

This is an interesting one, it creates a dump of the goroutine call stack and the number of running goroutines. Let’s take a look at the profile of the dump we did, but before we do let’s generate the dump.

$ ./pprof &
ab -k -c 80 -n 1000000 "http://127.0.0.1:6060/concat/?str=test&count=5000"

So in a different terminal we need to run at the same time, this is usually quite quick to create a goroutine profile unlike the CPU profile.

$ go tool pprof pprof http://127.0.0.1:6060/debug/pprof/goroutines

After it finishes dumping you can examine it when you need using the command

$ go tool pprof -http=:8080 dumps/pprof.pprof.goroutine.001.pb.gz

Graph

Flame-graph

If we take a look at both the graph and flamegraph, we can see that the biggest time we spend is something called runtime.gopark, if we take a look at the code comments of what this function does, we can see

// Puts the current goroutine into a waiting state and calls unlockf.
// If unlockf returns false, the goroutine is resumed.
// unlockf must not access this G's stack, as it may be moved between
// the call to gopark and the call to unlockf.
// Reason explains why the goroutine has been parked.
// It is displayed in stack traces and heap dumps.
// Reasons should be unique and descriptive.
// Do not re-use reasons, add new ones.

This is the goroutine scheduler, which happens because we spawn a lot of goroutines to serve every request, we should maybe think about limiting the amount of requests that we serve.

Block profile

It show function calls that led to blocking on synchronization primitives like mutexes and channels. One thing to remember is that before dumping the profile we have to SetBlockProfileRate in the application either in the main or the init function. We need to do this so we can capture the profiling information.

Contentions

In the picture we can see a lot of the time is spend in sync.Cond which is a conditional variable, with a an associated locker (often a *Mutex or a *RWMutex), which is held when changing the condition and when calling the Wait

Mean delay

In this picture we can see the mean delay for each block, which for sync.(*Cond).Wait is 2.65ms and sync.(*Mutex).Lock is 3.91ms.

Total delay

In this picture we can see the total time spent in each block, and in our case the sync.(*Cond).Wait is the problem because it’s being called 2614 times which brings to total time 6.91s.

Trace

And last but not least it’s the tracing tool, which visualizes the runtime events in fine detail over a run of a your application.

But I should also note that we shouldn’t use trace if you want to track down slow functions, or find where exactly is your program is spending most of its CPU time. For problems like that you should use pprof. trace is useful if you want to find out what is your program doing over time.

Let’s dig in a bit with the tracing now, to be able to trace we have to create a tracing file.

We can download the trace file just by doing executing the command bellow, and in the 10 seconds we need to generate some load to have some meaningful traces

wget http://localhost:6060/debug/pprof/trace?seconds=10 -O dumps/trace
go tool trace dumps/trace

We can see here what was executed at what point and what exactly was the flow between each piece in the system.

There are various other tools that trace has that can be of use to us like.

View trace
Goroutine analysis
Network blocking profile
Synchronization blocking profile
Syscall blocking profile
Scheduler latency profile
User-defined tasks
User-defined regions

Comparison of different profiles

We can also compare several profiles and visualise the differences between them, this is applicable to any pprof profile (except maybe trace), so we can compare the details between each heap profile capture by using the following commands.

go tool pprof -http=:8080 --base dumps/heap-profile-cmp-001.pb.gz dumps/heap-profile-cmp-002.pb.gz
go tool pprof -http=:8080 --base dumps/heap-profile-cmp-001.pb.gz dumps/heap-profile-cmp-003.pb.gz

And we can see the differences between both profiles to see how our application is behaving.

And that’s all folks. This was a lot of information and hopefully it will be of use to someone.

If you have some comments or remarks, let me know in the comment section.

Exposing metrics in your golang application

Sun, 30 Sep 2018 06:31:31 +0000

A lot of times it’s useful to expose metrics for your application so you can know what is going on with the application, there are two ways you can do this one it’s either using a pull or a push mechanism.

Golang has a package called expvar that provides a standardized interface to public variables, such as operation counters in servers, and exports these variables via HTTP in a JSON format.

We can use this package to export all our metrics, custom variables, counters or anything we would want and then visualize our data so we can analyze it and understand how our application works. The package has several types that it exports like float, int, string, map, and we can even define custom types to our metrics by implementing the Var interface

Let’s take a look at a simple implementation of how we can use expvar to export various metrics for our application, and even implementing a custom type that we can use in our application.

package main

import (
	"expvar"
	"fmt"
	"math/rand"
	"net/http"
	"os"
	"runtime"
	"time"
)

type TimeVar struct{ v time.Time }

func (o *TimeVar) Set(date time.Time)         { o.v = date }
func (o *TimeVar) Add(duration time.Duration) { o.v = o.v.Add(duration) }
func (o *TimeVar) String() string             { return fmt.Sprintf('"%s"', o.v.Format(time.RFC3339)) }

func NewStats(name string) *expvar.Map {
	stats = expvar.NewMap(name)
	stats.Set("counter", new(expvar.Int))
	stats.Set("success_rate", new(expvar.Float))
	stats.Set("pid", new(expvar.Int))

	go func() {
		t := time.NewTicker(time.Millisecond)
		for {
			select {
			case <-t.C:
				stats.Add("counter", rand.Int63n(100))
				stats.Get("success_rate").(*expvar.Float).Set(rand.Float64())
			}
		}
	}()

	return stats
}

var stats *expvar.Map
var lastUpdate *TimeVar

func init() {
	stats = NewStats("stats")
	lastUpdate = &TimeVar{}
	lastUpdate.Set(time.Now())
	stats.Get("pid").(*expvar.Int).Set(int64(os.Getpid()))
	expvar.Publish("last_update", lastUpdate)
	expvar.Publish("goroutines", expvar.Func(func() interface{} {
		return fmt.Sprintf("%d", runtime.NumGoroutine())
	}))
	expvar.Publish("cgocall", expvar.Func(func() interface{} {
		return fmt.Sprintf("%d", runtime.NumCgoCall())
	}))
	expvar.Publish("cpu", expvar.Func(func() interface{} {
		return fmt.Sprintf("%d", runtime.NumCPU())
	}))
}

func randomGoRoutineSpawn() {

	for {
		go func() {
			r := rand.Intn(10)
			time.Sleep(time.Duration(r) * time.Microsecond)
		}()
		r := rand.Intn(3)
		time.Sleep(time.Duration(r) * time.Microsecond)
	}
}

func main() {
	go randomGoRoutineSpawn()
	http.ListenAndServe(":6000", http.DefaultServeMux)

}

You may have noticed that we are using expvar.Func with Publish, what this one means is that the function will be called anytime we are retrieving the metrics, which makes it perfect for calculating the values only when they are needed and not all the time.

After running the application and querying the endpoint we have the following data.

$ curl localhost:6000/debug/vars

{
 "cgocall": "1",
 "cmdline": ["/tmp/go-build381034895/b001/exe/main"],
 "cpu": "4",
 "goroutines": "8",
 "last_update": "2018-09-30T09:59:08+02:00",
 "memstats": {"Alloc":2729112,"TotalAlloc":67979672,"Sys":72349944,"Lookups":0,"Mallocs":1055537,"Frees":1016347,"HeapAlloc":2729112,"HeapSys":66191360,"HeapIdle":62775296,"HeapInuse":3416064,"HeapReleased":0,"HeapObjects":39190,"StackInuse":917504,"StackSys":917504,"MSpanInuse":64752,"MSpanSys":98304,"MCacheInuse":6912,"MCacheSys":16384,"BuckHashSys":1442870,"GCSys":2445312,"OtherSys":1238210,"NextGC":4194304,"LastGC":1538294385797517989,"PauseTotalNs":1435570,"PauseNs":[79199,42451,50945,155414,71632,34360,112997,78210,80689,94101,197658,110553,47645,97254,37034,58762,86666,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"PauseEnd":[1538294350363250463,1538294352540250881,1538294354731420295,1538294356927816598,1538294358983245342,1538294361184114356,1538294363384264942,1538294365590777905,1538294367789220774,1538294370179305387,1538294372387563780,1538294374656050641,1538294376896879760,1538294379162055815,1538294381539608947,1538294383722948035,1538294385797517989,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"NumGC":17,"NumForcedGC":0,"GCCPUFraction":0.00008913387405867569,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":59,"Frees":24},{"Size":16,"Mallocs":579,"Frees":180},{"Size":32,"Mallocs":133,"Frees":72},{"Size":48,"Mallocs":163,"Frees":48},{"Size":64,"Mallocs":1053800,"Frees":1015659},{"Size":80,"Mallocs":25,"Frees":9},{"Size":96,"Mallocs":60,"Frees":8},{"Size":112,"Mallocs":11,"Frees":8},{"Size":128,"Mallocs":22,"Frees":12},{"Size":144,"Mallocs":7,"Frees":6},{"Size":160,"Mallocs":18,"Frees":5},{"Size":176,"Mallocs":8,"Frees":3},{"Size":192,"Mallocs":8,"Frees":6},{"Size":208,"Mallocs":39,"Frees":16},{"Size":224,"Mallocs":5,"Frees":3},{"Size":240,"Mallocs":0,"Frees":0},{"Size":256,"Mallocs":25,"Frees":7},{"Size":288,"Mallocs":11,"Frees":8},{"Size":320,"Mallocs":2,"Frees":1},{"Size":352,"Mallocs":21,"Frees":14},{"Size":384,"Mallocs":223,"Frees":0},{"Size":416,"Mallocs":8,"Frees":4},{"Size":448,"Mallocs":1,"Frees":0},{"Size":480,"Mallocs":0,"Frees":0},{"Size":512,"Mallocs":28,"Frees":22},{"Size":576,"Mallocs":8,"Frees":6},{"Size":640,"Mallocs":3,"Frees":1},{"Size":704,"Mallocs":2,"Frees":1},{"Size":768,"Mallocs":2,"Frees":0},{"Size":896,"Mallocs":16,"Frees":4},{"Size":1024,"Mallocs":2,"Frees":1},{"Size":1152,"Mallocs":8,"Frees":7},{"Size":1280,"Mallocs":1,"Frees":1},{"Size":1408,"Mallocs":1,"Frees":1},{"Size":1536,"Mallocs":1,"Frees":0},{"Size":1792,"Mallocs":6,"Frees":2},{"Size":2048,"Mallocs":5,"Frees":3},{"Size":2304,"Mallocs":7,"Frees":3},{"Size":2688,"Mallocs":2,"Frees":1},{"Size":3072,"Mallocs":2,"Frees":1},{"Size":3200,"Mallocs":0,"Frees":0},{"Size":3456,"Mallocs":0,"Frees":0},{"Size":4096,"Mallocs":14,"Frees":9},{"Size":4864,"Mallocs":9,"Frees":9},{"Size":5376,"Mallocs":1,"Frees":0},{"Size":6144,"Mallocs":4,"Frees":3},{"Size":6528,"Mallocs":2,"Frees":0},{"Size":6784,"Mallocs":0,"Frees":0},{"Size":6912,"Mallocs":0,"Frees":0},{"Size":8192,"Mallocs":1,"Frees":0},{"Size":9472,"Mallocs":4,"Frees":0},{"Size":9728,"Mallocs":0,"Frees":0},{"Size":10240,"Mallocs":0,"Frees":0},{"Size":10880,"Mallocs":0,"Frees":0},{"Size":12288,"Mallocs":0,"Frees":0},{"Size":13568,"Mallocs":0,"Frees":0},{"Size":14336,"Mallocs":0,"Frees":0},{"Size":16384,"Mallocs":0,"Frees":0},{"Size":18432,"Mallocs":0,"Frees":0},{"Size":19072,"Mallocs":0,"Frees":0}]},
 "stats": {"counter": 1893889, "pid": 23657, "success_rate": 0.45610231133193085}
}

Now we need to store this data somewhere, and we can visualize it and use the data in a lot of useful ways for us.

If you want to try it out you can pull the code from GitHub and run it in a container to see it in action

$ git clone https://github.com/ilijamt/gomeetupamsoct2018
$ cd watch
$ docker-compose up -d
$ export EXPVAR_HOST=http://`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' expvar_app_1`:6000/debug/vars
$ watch -n 1 curl $EXPVAR_HOST -s

This is one of the ways we can do the export of the metrics, another one is to use Prometheus, which I will cover in a future article, with the same example just to see how to implement it in Prometheus way.

Sum Two Values

Wed, 12 Sep 2018 09:27:31 +0000

Given an array of integers and a value, determine if there are any two integers in the array which sum equal to the given value.

Let’s see an example array that we can visualise it so we can come up with a proper approach.

[ 5, 6, 7, 1, 4, 2, 3 ]

We need to see if any 2 numbers create a sum of X. Let’s evaluate on a real case.

Sum	Values
10	6+4
11	5+6, 7+3

How we can solve this, we can use a hash set to store the values that can when added total to X, and we should also sort the array.

Algorithm

Scan the whole array once and store the visit elements in a hash set.
While scanning, for every element in the array we check if X - e is already visited
- If X - e is found in the hash set, it means that there is a pair(e, X - e) in array whose sum is equal to the given value
- If we have scanned all elements in the array and we didn’t find anything, return false

Solution with a hash set

let find_sum_of_two = function(A, X) {
    let values = new Set();
    for (let a in A) {
        if (values.has(X - A[a])) {
            return true;
        }
        values.add(A[a]);
    }
    return false;
}

let arr = [ 5, 6, 7, 1, 4, 2, 3 ];

console.log(arr, "has items with sum of", 5, find_sum_of_two(arr, 5));
console.log(arr, "has items with sum of", 10, find_sum_of_two(arr, 10));
console.log(arr, "has items with sum of", 11, find_sum_of_two(arr, 11));
console.log(arr, "has items with sum of", 20, find_sum_of_two(arr, 20));

And it should output

[ 5, 6, 7, 1, 4, 2, 3 ] 'has items with sum of' 5 true
[ 5, 6, 7, 1, 4, 2, 3 ] 'has items with sum of' 10 true
[ 5, 6, 7, 1, 4, 2, 3 ] 'has items with sum of' 11 true
[ 5, 6, 7, 1, 4, 2, 3 ] 'has items with sum of' 20 false

Runtime complexity: Linear O(n)

Memory complexity: O(n)

Solution

Let’s look at a solution that will allow us to improve a bit on the previous solution.

let find_sum_of_two = function(A, X) {
    let i = 0;
    let j = A.length - 1;

    while (i < j) {
        let sum = A[i] + A[j];
        if (sum == X) {
            return true;
        }

        if (sum < X) {
            i++;
        } else {
            j--;
        }
    }

    return false;

}

let arr = [1, 2, 3, 4, 5, 6, 7];

console.log(arr, "has items with sum of", 5, find_sum_of_two(arr, 5));
console.log(arr, "has items with sum of", 10, find_sum_of_two(arr, 10));
console.log(arr, "has items with sum of", 11, find_sum_of_two(arr, 11));
console.log(arr, "has items with sum of", 20, find_sum_of_two(arr, 20));

And it should output

[1, 2, 3, 4, 5, 6, 7] 'has items with sum of' 5 true
[1, 2, 3, 4, 5, 6, 7] 'has items with sum of' 10 true
[1, 2, 3, 4, 5, 6, 7] 'has items with sum of' 11 true
[1, 2, 3, 4, 5, 6, 7] 'has items with sum of' 20 false

Runtime complexity: Linear O(n), if we need to sort the array before we start then the complexity is O(nlogn)

Memory complexity: O(1)

Merge Overlapping Intervals

Wed, 12 Sep 2018 06:27:31 +0000

Given an array (list) of intervals as input where each interval has a start and end timestamps. Input array is sorted by starting timestamps. You are required to merge overlapping intervals and return output array (list).

Consider below input array. Intervals (2, 10), (4, 12), (11, 13), (15, 20) are overlapping so should be merged to one big interval (2, 13). Similarly interval (15, 20) doesn’t overlap anywhere so it should be added to the merged overlapping interval list.

The problem can be solved with a simple linear scan. And the intervals are sorted by the starting timestamps.

We can also expand this solution for an unsorted array of intervals, by just sorting the intervals based on the start time.

Algorithm

List of input intervals is given, and we we keep the merged inteval list in a separate list
For each interval in the list
- If interval is overlapping with the last interval in the output list we merge these two intervals and update the last interval of the output interval with the merged one
- Otherwise we add the input interval to the output interval

Solution

let merge_intervals = function(v1) {
  if (!v1 || v1.length === 0) {
    return;
  }

  let v2 = [];
  v2.push([ v1[0][0], v1[0][1] ]);

  for (let i = 0; i < v1.length; i++) {
    let x1 = v1[i][0];
    let y1 = v1[i][1];
    let x2 = v2[v2.length - 1][0];
    let y2 = v2[v2.length - 1][1];

    if (y2 >= x1) {
      v2[v2.length - 1][1] = Math.max(y1, y2);
    } else {
      v2.push([ x1, y1 ]);
    }

  }
  return v2;
};

console.log(merge_intervals([[1,5],[3,7],[4,6],[6,8],[10,12],[11,15]]));
console.log(merge_intervals([[4,12],[13,16],[19,20],[20,24]]));
console.log(merge_intervals([[2,10],[4, 12], [11, 13], [15, 20]]));

And the output would be

[ [ 1, 8 ], [ 10, 15 ] ]
[ [ 4, 12 ], [ 13, 16 ], [ 19, 24 ] ]
[ [ 2, 12 ] ]

Runtime complexity: Linear O(n), if we need to sort the array before we start then the complexity is O(nlogn)

Memory complexity: O(n) worst case, when they are no overlapping intervals

Finding maximum in a sliding window

Mon, 10 Sep 2018 14:55:31 +0000

So I’m trying to stay true to my resolution, write one or more article about algorithms and datastructure everyday to refresh my memory on how everything works.

Description

Given a large array of integers and a window of size ‘w’, find the current maximum in the window as the window slides through the entire array.

Example

Consider the array below and let’s try to find all maximum with window size = 3, for explanation let’s start with a smaller array so we can go over each steps.

[-1 2 -2 5 9]

For the first window size (3) elements, the max is 2
Slide window one position right and then the max element is 5
Slide window one position right again, and then the max element is 9

Solutions

Algorithm

Check if the window size is bigger than the array it self, and if so return
Find the max for the first window, by storing the max value into a window
Push the first element of the window into the results array
Iterate the rest of the elements ** Remove all the elements from the tail of the window that are equal or smaller than the current element ** Remove the element in the head if the index no longer is in the current window ** Push the current element into the window ** Current max is at the head of the array and can be pushed in results

Iterative solution

Runtime complexity

Linear, О(nlogw)

Memory complexity

Logarithmic, О(w)

The reason why it has O(logn) memory complexity is because it will have to consume memory on the stack.

Implementation

let find_max_sliding_window = function(arr, window_size) {
  let result = [];
  if (window_size > arr.length) {
    return [];
  }

  let wind = [];
  for (let i = 0; i < window_size; i++) {
    while (wind.length > 0 && arr[i] >= arr[wind[wind.length - 1]]) {
      wind.pop(); // remove the element of the window
    }
    wind.push(i); // put the index of the element that has max in the window
  }

  result.push(arr[wind[0]]);

  // move the window to the right side and go over the window again
  for (let i = window_size; i < arr.length; i++) {
    while (wind.length > 0 && arr[i] >= arr[wind[wind.length - 1]]) {
      wind.pop(); // remove the element of the window
    }
    // move the element to right and remove the first element in the window
    if (wind.length > 0 && (wind[0] <= i - window_size)) {
      wind.shift();
    }
    wind.push(i);
    result.push(arr[wind[0]]);
  }

  return result;
};

console.log(find_max_sliding_window([97, 7, 36, 16, 90, 54, 54, 81, 43, 66, 72, 97, 68, 27, 12, 83, 8, 26, 70, 8], 2));
console.log(find_max_sliding_window([97, 7, 36, 16, 90, 54, 54, 81, 43, 66, 72, 97, 68, 27, 12, 83, 8, 26, 70, 8], 3));
console.log(find_max_sliding_window([97, 7, 36, 16, 90, 54, 54, 81, 43, 66, 72, 97, 68, 27, 12, 83, 8, 26, 70, 8], 4));
console.log(find_max_sliding_window([97, 7, 36, 16, 90, 54, 54, 81, 43, 66, 72, 97, 68, 27, 12, 83, 8, 26, 70, 8], 5));

After running the application the output is

[97, 36, 36, 90, 90, 54, 81, 81, 66, 72, 97, 97, 68, 27, 83, 83, 26, 70, 70]
[97, 36, 90, 90, 90, 81, 81, 81, 72, 97, 97, 97, 68, 83, 83, 83, 70, 70]
[97, 90, 90, 90, 90, 81, 81, 81, 97, 97, 97, 97, 83, 83, 83, 83, 70]
[97, 90, 90, 90, 90, 81, 81, 97, 97, 97, 97, 97, 83, 83, 83, 83]

This was quite easy but we have to wrap our heads around how we would approach this, if we remember that do we really need to store all the elements in memory, and how can we create an efficient structure that can store the data for us.

Binary search algorithm, find the index of a given key

Sun, 09 Sep 2018 17:45:35 +0000

It’s been a while since I’ve been writing on the blog, so I’ve decided to start being more active and since it’s been almost 15 years since I’ve studied this in University, it can also be a good refreshment course for me as well as anyone else looking into refreshing their memory or learning.

Binary search is a very useful algorithm, also known as half-interval search, logarithmic search, or binary chop, is a search algorithm that finds the position of a target value within a sorted array.

Binary search is used to find the index of an element in a sorted array, and if the element doesn’t exist, that can be determined efficiently as well. It does it by dividing the input array by half at every step, and after every step, either we have found the index that we are looking for or half of the array can be discarded.

Description

Given a sorted array of integers, return the index of the given key. Return -1 if not found.

Example

Given the following array, if search key is 4, binary search will return 2.

Solutions

There are a couple of ways to solve this problem easily, we can either use a recursive or iterative approach, but before we look into both let’s evaluate the complexity for both solutions, and the algorithms for both solutions.

Algorithm

At every step, consider the array between low and high indices
Calculate the middle index.
If element at the middle index is the key, return middle.
If element at middle is greater than the key, then reduce the array size such that high becomes mid - 1. Index at low remains the same.
If element at middle is less than the key, then reduce the array size such that low becomes mid + 1. Index at high remains the same.
When low is greater than high, key doesn’t exist. Return -1.

Recursive

Runtime complexity

Logarithmic, О(logn)

Memory complexity

Logarithmic, О(logn)

The reason why it has O(logn) memory complexity is because it will have to consume memory on the stack.

Implementation

#include <iostream>
using namespace std;

int binary_search_recursive(int A[], int key, int low, int high) {
  if (low > high) {
    return -1;
  }

  int mid = low + ((high - low) / 2);
  if (A[mid] == key) {
    return mid;
  } else if (key < A[mid]) {
    return binary_search_recursive(A, key, low, mid - 1);
  }

  return binary_search_recursive(A, key, mid + 1, high);
}

int main(int argc, char ** argv) {
  int arr[] = { 1, 3, 4, 6, 7, 8, 10, 13, 14 };
  int size = sizeof(arr) / sizeof(arr[0]);
  cout << "Key(2) found at: " << binary_search_recursive(arr, 2, 0, size - 1) << endl;
  cout << "Key(4) found at: " << binary_search_recursive(arr, 4, 0, size - 1) << endl;
  cout << "Key(9) found at: " << binary_search_recursive(arr, 9, 0, size - 1) << endl;
  cout << "Key(13) found at: " << binary_search_recursive(arr, 13, 0, size - 1) << endl;
  cout << "Key(90) found at: " << binary_search_recursive(arr, 90, 0, size - 1) << endl;
  return 0;
}

After running the application the output is

$ g++ arr-binary-search-rec.cpp
$ ./a.out
Key(2) found at: -1
Key(4) found at: 2
Key(9) found at: -1
Key(13) found at: 7
Key(90) found at: -1

Iterative

Runtime complexity

Logarithmic О(logn)

Memory complexity

Constants, O(1)

Implementation

#include <iostream>
using namespace std;

int binary_search_iterative(int A[], int key, int len) {
  int low = 0;
  int high = len - 1;

  while (low <= high) {
    int mid = low + ((high - low) / 2);
    if (A[mid] == key) {
      return mid;
    } else if (key < A[mid]) {
      high = mid - 1;
    } else {
      low = mid + 1;
    }
  }

  return -1;
}

int main(int argc, char ** argv) {
  int arr[] = { 1, 3, 4, 6, 7, 8, 10, 13, 14 };
  int size = sizeof(arr) / sizeof(arr[0]);
  cout << "Key(2) found at: " << binary_search_iterative(arr, 2, size - 1) << endl;
  cout << "Key(4) found at: " << binary_search_iterative(arr, 4, size - 1) << endl;
  cout << "Key(9) found at: " << binary_search_iterative(arr, 9, size - 1) << endl;
  cout << "Key(13) found at: " << binary_search_iterative(arr, 13, size - 1) << endl;
  cout << "Key(90) found at: " << binary_search_iterative(arr, 90, size - 1) << endl;
  return 0;
}

After running the application the output is

$ g++ arr-binary-search-iter.cpp
$ ./a.out
Key(2) found at: -1
Key(4) found at: 2
Key(9) found at: -1
Key(13) found at: 7
Key(90) found at: -1

Resources

Picture for binary search taken from Wikipedia
If you want to know about different variations of binary search take a look at the wikipedia page

Site to site with OpenVPN with TLS and FirewallD on Debian

Mon, 05 Feb 2018 21:02:58 +0000

So I needed to setup a site to site transport between our two datacenters, so our internal network will be able to communicate between the two datacenters easily.

There are a lot of ways to setup site to site VPN link between two networks, in our case let’s take a look at one way using OpenVPN.

This is quite long tutorial so get yourself a bit comfortable so we can get started.

Our network looks like this

Datacenter at location A has internal network 10.5.200.20 on the interface eth1 and 94.x.x.x.x on eth0, we will configure the openvpn server on this location. Datacenter at location B has internal network 10.0.0.5 on the interface eth1 and 93.x.x.x.x on eth0.

Both machine have two network interfaces eth0 which is connected to the internet, and eth1 which is connected the the LAN

The IP subnet has to be different for this to work, if they are on the same subnet this won’t work.

VPN gateway at location A

So to begin login to the machine in the location A

Forwarding

$ cat <<EOF > /etc/sysctl.d/30-openvpn-forward.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF
$ sysctl -p /etc/sysctl.d/30-openvpn-forward.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1

Installation

apt-get install openvpn easy-rsa firewalld

Firewall

Let’s configure the firewall to allow us to connect to the machine and all the necessary configuration needed for it

$ firewall-cmd --permanent --zone=public --add-source=93.x.x.x/32
$ firewall-cmd --permanent --zone=trusted --add-interface=eth1
$ firewall-cmd --permanent --zone=trusted --add-interface=tun0
$ firewall-cmd --permanent --zone=trusted --add-masquerade
$ DEV=$(ip route get 8.8.8.8 | awk 'NR==1 {print $(NF-2)}')
$ firewall-cmd --permanent --direct --passthrough ipv4 -t nat -A POSTROUTING -s  172.16.1.0/24 -o $DEV -j MASQUERADE
$ firewall-cmd --reload

Generating OpenVPN keys

After we have installed all this we need to generate some configuration and tls keys for us to use.

$ make-cadir ~/ca
$ cd ~/ca
$ ls -lah
drwx------ 2 root root 4.0K Feb  5 21:23 .
drwxr-xr-x 4 root root 4.0K Feb  5 21:23 ..
lrwxrwxrwx 1 root root   28 Feb  5 21:23 build-ca -> /usr/share/easy-rsa/build-ca
lrwxrwxrwx 1 root root   28 Feb  5 21:23 build-dh -> /usr/share/easy-rsa/build-dh
lrwxrwxrwx 1 root root   31 Feb  5 21:23 build-inter -> /usr/share/easy-rsa/build-inter
lrwxrwxrwx 1 root root   29 Feb  5 21:23 build-key -> /usr/share/easy-rsa/build-key
lrwxrwxrwx 1 root root   34 Feb  5 21:23 build-key-pass -> /usr/share/easy-rsa/build-key-pass
lrwxrwxrwx 1 root root   36 Feb  5 21:23 build-key-pkcs12 -> /usr/share/easy-rsa/build-key-pkcs12
lrwxrwxrwx 1 root root   36 Feb  5 21:23 build-key-server -> /usr/share/easy-rsa/build-key-server
lrwxrwxrwx 1 root root   29 Feb  5 21:23 build-req -> /usr/share/easy-rsa/build-req
lrwxrwxrwx 1 root root   34 Feb  5 21:23 build-req-pass -> /usr/share/easy-rsa/build-req-pass
lrwxrwxrwx 1 root root   29 Feb  5 21:23 clean-all -> /usr/share/easy-rsa/clean-all
lrwxrwxrwx 1 root root   33 Feb  5 21:23 inherit-inter -> /usr/share/easy-rsa/inherit-inter
lrwxrwxrwx 1 root root   24 Feb  5 21:23 keys -> /usr/share/easy-rsa/keys
lrwxrwxrwx 1 root root   28 Feb  5 21:23 list-crl -> /usr/share/easy-rsa/list-crl
-rw-r--r-- 1 root root 7.7K Feb  5 21:23 openssl-0.9.6.cnf
-rw-r--r-- 1 root root 8.3K Feb  5 21:23 openssl-0.9.8.cnf
-rw-r--r-- 1 root root 8.2K Feb  5 21:23 openssl-1.0.0.cnf
-rw-r--r-- 1 root root 8.2K Feb  5 21:23 openssl.cnf
lrwxrwxrwx 1 root root   27 Feb  5 21:23 pkitool -> /usr/share/easy-rsa/pkitool
lrwxrwxrwx 1 root root   31 Feb  5 21:23 revoke-full -> /usr/share/easy-rsa/revoke-full
lrwxrwxrwx 1 root root   28 Feb  5 21:23 sign-req -> /usr/share/easy-rsa/sign-req
-rw-r--r-- 1 root root 2.1K Feb  5 21:23 vars
lrwxrwxrwx 1 root root   35 Feb  5 21:23 whichopensslcnf -> /usr/share/easy-rsa/whichopensslcnf

You can replace the openssl.cnf with whichever version you want to use for generation, I decided to go with openssl-1.0.0.cnf

Open the file ~/ca/vars and update accordingly, you can customize it however you want but you should at least modify the following to suit your organization.

export KEY_COUNTRY="US"
export KEY_PROVINCE="CA"
export KEY_CITY="SanFrancisco"
export KEY_ORG="Fort-Funston"
export KEY_EMAIL="me@myhost.mydomain"
export KEY_OU="MyOrganizationalUnit"

After you have edited the file you need to source it so we have the environment variables ready, to do so you can just run

$ cd ~/ca
$ . ./vars
$ env | grep ^KEY
KEY_OU=MyOrganizationalUnit
KEY_SIZE=2048
KEY_NAME=EasyRSA
KEY_EXPIRE=3650
KEY_EMAIL=me@myhost.mydomain
KEY_ORG=Fort-Funston
KEY_CITY=SanFrancisco
KEY_COUNTRY=US
KEY_PROVINCE=CA
KEY_CONFIG=/root/ca/openssl.cnf
KEY_DIR=/root/ca/keys
$ ./clean-all

Now you are ready to build your ca key and tls-crypt key for OpenVPN

$ cd ~/ca
$ ./build-ca
$ openvpn --genkey --secret ~/ca/keys/ta.key

You CA is ready now we have to generate a server key, make sure you sign the certificate when asked

$ cd ~/ca
$ ./build-key-server server
$ ./build-dh

And we also need to generate a client key for each client that will connect, make sure you sign the certificates when asked

$ cd ~/ca
$ ./build-key locb

Now let’s see what have we generated your keys folder should have the following files

$ ls -lh ~/ca/keys
total 84K
-rw-r--r-- 1 root root 5.4K Feb  5 21:31 01.pem
-rw-r--r-- 1 root root 5.3K Feb  5 21:35 02.pem
-rw-r--r-- 1 root root 1.7K Feb  5 21:30 ca.crt
-rw------- 1 root root 1.7K Feb  5 21:30 ca.key
-rw-r--r-- 1 root root  424 Feb  5 21:33 dh2048.pem
-rw-r--r-- 1 root root  238 Feb  5 21:35 index.txt
-rw-r--r-- 1 root root   21 Feb  5 21:35 index.txt.attr
-rw-r--r-- 1 root root   21 Feb  5 21:31 index.txt.attr.old
-rw-r--r-- 1 root root  120 Feb  5 21:31 index.txt.old
-rw-r--r-- 1 root root 5.3K Feb  5 21:35 locb.crt
-rw-r--r-- 1 root root 1.1K Feb  5 21:35 locb.csr
-rw------- 1 root root 1.7K Feb  5 21:35 locb.key
-rw-r--r-- 1 root root    3 Feb  5 21:35 serial
-rw-r--r-- 1 root root    3 Feb  5 21:31 serial.old
-rw-r--r-- 1 root root 5.4K Feb  5 21:31 server.crt
-rw-r--r-- 1 root root 1.1K Feb  5 21:31 server.csr
-rw------- 1 root root 1.7K Feb  5 21:31 server.key
-rw------- 1 root root  636 Feb  5 21:30 ta.key

Now we have everything so we can send finally do the connection between our datacenters.

Configuration

So let’s prepare some things before we start

$ mkdir -p /etc/openvpn/ccd
$ cp -v ~/ca/keys/{ca.crt,server.*,dh2048.pem,ta.key} /etc/openvpn/server
'/root/ca/keys/ca.crt' -> '/etc/openvpn/server/ca.crt'
'/root/ca/keys/server.crt' -> '/etc/openvpn/server/server.crt'
'/root/ca/keys/server.csr' -> '/etc/openvpn/server/server.csr'
'/root/ca/keys/server.key' -> '/etc/openvpn/server/server.key'
'/root/ca/keys/dh2048.pem' -> '/etc/openvpn/server/dh2048.pem'
'/root/ca/keys/ta.key' -> '/etc/openvpn/server/ta.key'

And now lets make our configuration for the server. We need to create a separate user for our process to run in, so it’s isolated from the system.

$ addgroup ovpn
$ useradd --shell=/bin/false -g ovpn ovpn
$ cat /etc/openvpn/server.conf
dev tun
persist-key
persist-local-ip
persist-remote-ip
persist-tun
topology subnet
port 1194
proto udp
keepalive 10 60

ca /etc/openvpn/server/ca.crt
cert /etc/openvpn/server/server.crt
key /etc/openvpn/server/server.key
dh /etc/openvpn/server/dh2048.pem

server 172.16.1.0 255.255.255.0
client-to-client

client-config-dir ccd

explicit-exit-notify 10

user ovpn
group ovpn

tls-crypt /etc/openvpn/server/ta.key
auth SHA512
tls-version-min 1.2
tls-cipher TLS-DHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256
ncp-ciphers AES-256-GCM:AES-256-CBC

ifconfig-pool-persist ipp.txt
status openvpn-status.log
log /var/log/openvpn.log
verb 4

route 10.0.0.0 255.255.255.0

tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
reneg-sec 0

cipher AES-128-CBC

All the configuration in here is self explanatory, the only thing worth of not is route 10.0.0.0 255.255.255.0, which needs to be in the main file so the server knows that it needs to route that subnet to the connected client, if you don’t add that line then we will be missing the required routes to access the server.

Now we need to setup the vm in the location b, we still have to install the same packages as for location a, but before we do that we need to create a configuration file for the client

$ cd /etc/openvpn/ccd
cat <<EOF > /etc/openvpn/ccd/locb
push "route 10.5.100.0 255.255.252.0"
iroute 10.0.0.0 255.255.255.0
EOF

This will tell that the client locb when connected that it needs to add the route for accessing the subnet in loca and for the server to setup an internal route for the locb

Service

Let’s enable the server

$ systemctl disable openvpn
$ systemctl enable openvpn@server
$ systemctl restart openvpn@server
$ systemctl status openvpn@server.service

VPN gateway at location B

Now we are ready to connect we need to transfer the files ca.crt,locb.*,dh2048.pem,ta.key in the keys folder from the server to the client in /etc/openvpn/client folder.

Forwarding

$ cat <<EOF > /etc/sysctl.d/30-openvpn-forward.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF
$ sysctl -p /etc/sysctl.d/30-openvpn-forward.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1

Installation

Let’s install the prerequisites

$ apt-get install openvpn firewalld
$ addgroup ovpn
$ useradd --shell=/bin/false -g ovpn ovpn

Firewall

Let’s configure the firewall to allow us to connect to the machine and all the necessary configuration needed for it

$ firewall-cmd --permanent --zone=public --add-source=94.x.x.x/32
$ firewall-cmd --permanent --zone=trusted --add-interface=eth1
$ firewall-cmd --permanent --zone=trusted --add-interface=tun0
$ firewall-cmd --permanent --zone=trusted --add-masquerade
$ DEV=$(ip route get 8.8.8.8 | awk 'NR==1 {print $(NF-2)}')
$ firewall-cmd --permanent --direct --passthrough ipv4 -t nat -A POSTROUTING -s  172.16.1.0/24 -o $DEV -j MASQUERADE
$ firewall-cmd --reload

Configuration

$ cat /etc/openvpn/client.conf
client
dev tun
persist-key
persist-tun
persist-local-ip
persist-remote-ip
proto udp
nobind
user ovpn
group ovpn
remote-cert-tls server
auth SHA512
verb 4

remote s2s.example.com 1194

ca /etc/openvpn/client/ca.crt
cert /etc/openvpn/client/locb.crt
key /etc/openvpn/client/locb.key
tls-crypt /etc/openvpn/client/ta.key

log /var/log/openvpn.log

keepalive 10 60

$ systemctl disable openvpn
$ systemctl enable openvpn@client
$ systemctl restart openvpn@client
$ systemctl status openvpn@client.service

Routes

The only thing missing now is setting up the routes required so we know where we need to go for a given subnet

/etc/network/interfaces

$ cat /etc/network/interfaces
source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

auto ens18
iface ens18 inet dhcp

auto ens19
iface ens19 inet static
    address 10.0.0.3
    netmask 255.255.255.0
    up route add -net 10.5.100.0 netmask 255.255.252.0 gw 10.0.0.5
    down route add -net 10.5.100.0 netmask 255.255.252.0 gw 10.0.0.5

So now everytime our interface is configured our route will be set up so we can access the servers on the other side of the VPN

ip route

We need to add a static route on the internal network interface so we can route the subnet.

$ ip route
default via 95.x.x.x dev ens18 onlink
10.0.0.0/24 dev ens19 proto kernel scope link src 10.0.0.3
95.x.x.x/25 dev ens18 proto kernel scope link src 95.x.x.x

$ ping 10.5.100.10 -c1
PING 10.5.100.10 (10.5.100.10) 56(84) bytes of data.

--- 10.5.100.10 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

$ ip route add 10.5.100.0/22 via 10.0.0.5 dev ens19

$ ip route
default via 95.x.x.x dev ens18 onlink
10.0.0.0/24 dev ens19 proto kernel scope link src 10.0.0.3
10.5.100.0/22 via 10.0.0.5 dev ens19
95.x.x.x/25 dev ens18 proto kernel scope link src 95.x.x.x

$ ping 10.5.100.10 -c1
PING 10.5.100.10 (10.5.100.10) 56(84) bytes of data.
64 bytes from 10.5.100.10: icmp_seq=1 ttl=62 time=2.17 ms

--- 10.5.100.10 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.171/2.171/2.171/0.000 ms

DHCP

If we use DHCP we can easily put a stateless route so all machines automatically get the correct route.

dnsmasq

We just need to add this to the DHCP server in location A and we are done, the route will be automatically setup when the client requests an IP address.

dhcp-option=121,10.5.100.0/22,10.0.0.5

Testing

And now let’s see the result of our process finally

VPN gateway in location A

$ traceroute 10.0.0.3 -n
traceroute to 10.0.0.3 (10.0.0.3), 30 hops max, 60 byte packets
 1  172.16.1.2  2.050 ms  2.038 ms  2.033 ms
 2  10.0.0.3  2.239 ms  2.242 ms  2.315 ms

VPN gateway in location B

$ traceroute 10.5.100.10 -n
traceroute to 10.5.100.10 (10.5.100.10), 30 hops max, 60 byte packets
 1  172.16.1.1  2.023 ms  2.007 ms  2.000 ms
 2  10.5.100.10  1.991 ms  1.986 ms  1.979 ms

Machine in location B to a machine in location A

$ traceroute 10.5.100.10 -n
traceroute to 10.5.100.10 (10.5.100.10), 30 hops max, 60 byte packets
 1  10.0.0.5  0.514 ms  0.497 ms  0.486 ms
 2  * * *
 3  10.5.100.10  2.238 ms  2.234 ms  2.219 ms

The 2nd hop is our VPN gateway doing the forwarding

Speed

The speed we got it’s not that bad, it’s 281 Mbits/sec which is around 35 MB/s. We can make it even faster by using Elliptic curve crypto (ECC) for the keys

Machine on location A

We start an iperf server on any machine on location A by logging into it and running

$ apt-get install iperf
$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.5.100.10 port 5001 connected with 10.5.100.10 port 42076
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   335 MBytes   280 Mbits/sec

Machine on location B

We start an iperf client on any machine on location B by logging into it and running

$ apt-get install iperf
$ iperf -c 10.5.100.10
------------------------------------------------------------
Client connecting to 10.5.100.20, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.3 port 42076 connected with 10.5.100.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   335 MBytes   281 Mbits/sec

Confirm before executing shutdown/reboot command on linux

Mon, 23 Oct 2017 11:54:00 +0000

I was rushing to leave and was still logged into a server so I wanted to shutdown my laptop, but what I didn’t notice is that I was still connected to the remote server. Luckily before pressing enter I noticed I’m not on my machine but on a remote server. So I was thinking there should be a very easy way to prevent it from happening again, to me or to anyone else.

So first thing we need to create a new bash script at /usr/local/bin/confirm with the contents bellow and with execution permissions

#!/usr/bin/env bash
echo "About to execute $1 command"
echo -n "Would you like to proceed y/n? "
read reply

if [ "$reply" = y -o "$reply" = Y ]
then
   $1 "${@:2}"
else
   echo "$1 ${@:2} cancelled"
fi

Now only thing left to do is to setup the aliases so they go through this command to confirm instead of directly calling the command.

So I create the following files

/etc/profile.d/confirm-shutdown.sh

alias shutdown="/usr/local/bin/confirm /sbin/shutdown"

/etc/profile.d/confirm-reboot.sh

alias reboot="/usr/local/bin/confirm /sbin/reboot"

Now when I actually try to do a shutdown/reboot it will prompt me like so.

ilijamt@x1 ~ $ reboot 
Before proceeding to perform /sbin/reboot, please ensure you have approval to perform this task
Would you like to proceed y/n? n
/sbin/reboot  cancelled

Compiling Redis Desktop Manager on Debian

Sun, 01 Jan 2017 21:20:24 +0000

So I’ve used redis desktop manager in the past, but suddenly I can’t use it anymore it’s not compiling on debian, and there are no packages for the application. So I decided to find a way to to compile it on my machine, which uses Debian Stretch.

So first we need to install some prerequisites so we can compile the application

sudo apt-get install build-essential git libssh2-1-dev zlib1g zlib1g-dev cmake libssl-dev qt5-default automake libtool \
    libssl-dev libssh2-1-dev g++ libgl1-mesa-dev qtdeclarative5-dev qml-module-qtgraphicaleffects \
    qml-module-qtquick-controls qml-module-qtquick-dialogs qml-module-qtquick-extras \
    qml-module-qtquick-layouts qml-module-qtquick-privatewidgets \
    qml-module-qtquick-window2 qml-module-qtquick2 qml-module-qttest

Now to compile we need to pull the code from github, currently the stable version is 0.8.8, so we can execute the following commands to pull the code from Github.

mkdir ~/builds && cd ~/builds
export RDM_ROOT=~/builds/rdm
export RDM_VERSION=0.8.8

git clone --recursive https://github.com/uglide/RedisDesktopManager.git --branch $RDM_VERSION --depth 1 $RDM_ROOT

Now that we have the code for Redis Desktop Manager, we need to pull some extra repos so we can compile the application. We need to pull and compile breakpad

git clone https://chromium.googlesource.com/linux-syscall-support $RDM_ROOT/3rdparty/gbreakpad/src/third_party/lss

cd $RDM_ROOT/3rdparty/gbreakpad/
touch README
./configure
make -j 4

Now we have everything required to compile Redis Desktop Manager, running the following commands will compile Redis Desktop Manager, so it can be used.

cd $RDM_ROOT/src

echo "#define RDM_VERSION \"$RDM_VERSION\"" > version.h

qmake
make -j 4

The binary is located in $RDM_ROOT/bin, depending on the compilation, in my case it was in $RDM_ROOT/bin/linux/release, so just copy the binary rdm to where ever you want and you can run it.

If you don’t want to polute your OS, you can always do this in a chroot.

debootstrap --arch amd64 stretch /srv/chroot/stretch http://ftp.debian.org/debian
chroot /srv/chroot/stretch

The rest of the steps are the same, just now you don’t need to install all the packages from above, but you will need to install all the qml packages on the machine you will run the application.

And after it finishes compiling, you can remove the folder by issuing.

rm -rf /srv/chroot/stretch

Now you have redis desktop manager. Well that’s it.

Enjoy.

Codility - OddOccurrencesInArray

Tue, 13 Dec 2016 17:12:03 +0000

A non-empty zero-indexed array A consisting of N integers is given. The array contains an odd number of elements, and each element of the array can be paired with another element that has the same value, except for one element that is left unpaired.

For example, in array A such that:

  A[0] = 9  A[1] = 3  A[2] = 9
  A[3] = 3  A[4] = 9  A[5] = 7
  A[6] = 9

the elements at indexes 0 and 2 have value 9
the elements at indexes 1 and 3 have value 3
the elements at indexes 4 and 6 have value 9
the element at index 5 has value 7 and is unpaired

Write a function that, given an array A consisting of N integers fulfilling the above conditions, returns the value of the unpaired element.

For example, given array A such that:

  A[0] = 9  A[1] = 3  A[2] = 9
  A[3] = 3  A[4] = 9  A[5] = 7
  A[6] = 9

the function should return 7, as explained in the example above.

Assume that:

N is an odd integer within the range [1..1,000,000]
each element of array A is an integer within the range [1..1,000,000,000]
all but one of the values in A occur an even number of times

Complexity:

expected worst-case time complexity is O(N);
expected worst-case space complexity is O(1), beyond input storage (not counting the storage required for input arguments).

Elements of input arrays can be modified.

Solution

This problem looks very complicated in the begining, but actually it’s quite simple.

We need to find out which number is not paired, and that can easily be accomplished by using bitwise operation, more to the point XOR operator.

0 0 = 0
0 A = A
A 0 = A
A A = 0

From this table we can see that everytime we use a XOR operator on the number it self it will reset to zero, that’s what we need.

So the solution is iterating the whole array and doing a XOR on a variable that will hold the missing number.

func Solution(A []int) int {

    var number int = 0;
    
    for _, v := range A {
        number ^= v
    }
    
    return number
}

Codility - CyclicRotation

Tue, 13 Dec 2016 16:32:03 +0000

A zero-indexed array A consisting of N integers is given. Rotation of the array means that each element is shifted right by one index, and the last element of the array is also moved to the first place.

For example, the rotation of array A = [3, 8, 9, 7, 6] is [6, 3, 8, 9, 7]. The goal is to rotate array A K times; that is, each element of A will be shifted to the right by K indexes.

Write a function that, given a zero-indexed array A consisting of N integers and an integer K, returns the array A rotated K times.

For example, given array A = [3, 8, 9, 7, 6] and K = 3, the function should return [9, 7, 6, 3, 8].

Assume that:

N and K are integers within the range [0..100];
each element of array A is an integer within the range [−1,000..1,000].

In your solution, focus on correctness. The performance of your solution will not be the focus of the assessment.

function solution(A, K) {

    var size = A.length;
    var ret = [];

    if (K < 0 || K > 100 || size == 0) {
        return ret;
    }

    if (size == 1) {
        return A;
    }

    for (i = 0; i < size; i++) {
        ret[(i + K) % size] = A[i];
    }

    return ret;
}

Some of the steps are obvious but let’s see the rest of them.

If we want to shift an element K times we can calculate the new index by using (index + shift_times) % size_of_element, but how does this actually work.

The modulo operator returns the reminder of the division between two numbers. So if we have an array with 10 elements, and we want to shift the index 3, 5 times if we do it manually we can figure out that the new index should be 8.

3 + 5 = 8
8 % 10 = 8

But lets say that the array has 5 elements instead of 10 then, we know that we need to shift the array to position 3, if you do it one step at a time you can verify that it’s so.

3 + 5 = 8
8 % 5 = 3

But what if the array had 3 elements

3 + 5 = 8
8 % 3 = 2

And there you go very simple.

Codility - BinaryGap

Tue, 13 Dec 2016 16:01:03 +0000

A binary gap within a positive integer N is any maximal sequence of consecutive zeros that is surrounded by ones at both ends in the binary representation of N.

For example, number 9 has binary representation 1001 and contains a binary gap of length 2. The number 529 has binary representation 1000010001 and contains two binary gaps: one of length 4 and one of length 3. The number 20 has binary representation 10100 and contains one binary gap of length 1. The number 15 has binary representation 1111 and has no binary gaps.

Write a function that, given a positive integer N, returns the length of its longest binary gap. The function should return 0 if N doesn’t contain a binary gap.

For example, given N = 1041 the function should return 5, because N has binary representation 10000010001 and so its longest binary gap is of length 5.

Assume that:

N is an integer within the range [1..2,147,483,647].

Complexity:

expected worst-case time complexity is O(log(N));
expected worst-case space complexity is O(1).

So lets see how we can solve this. Actually it’s quite an easy thing to do. But what would be the fastest way to do it.

There are several options to solve this case, one way would be to convert it to a binary string and explode the string to a group, and you can just check which group has the most zeroes. Another way and fastest way would be to do bit checks on the number it self, this is what we gonna try.

func Solution(N int) int {

	var max int = 0
	var current int = 0
	var startFound bool = false

	for i := 31; i >= 0; i-- {
		bit := ((N & (1 << uint(i))) > 0)
		if startFound {
			// we have found the start
			if !bit {
				current++
			} else {
				if max < current {
					max = current
				}
				current = 0
			}
		} else {
			startFound = bit
		}
	}

	return max

}

We can probably optimze the following code a bit more, but it should be sufficient for now, and I leave it up to you guys to see and decide how to do it.

Automatic starting and stopping of AWS EC2 instances with Lambda and CloudWatch

Sat, 19 Nov 2016 10:12:35 +0000

I needed to figure out a way to start/stop instances automatically during certain periods. The obvious way is Lambda, but how to do it. We wanted some instances to run from Monday to Friday, and to start at 7am and stop at 5pm.

Luckily there is a library that abstracts everything you need for starting and stopping your instances. And coupled with Lambda and CloudWatch we can easily accomplish what we want.

IAM Role

First thing let’s create a new IAM role. The role is defined bellow, so we need some things so we can create logs and allow our Lambda functions to be able to start/stop an instance, with the following steps.

Go to IAM Management console
Roles
Create New Role
- Role Name: lambda_start_stop_ec2
Role Type
- AWS Service Roles: AWS Lambda
Create role
Edit lambda_start_stop_ec2 role, and create a new custom inline policy with the json contents bellow.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:Start*",
        "ec2:Stop*"
      ],
      "Resource": "*"
    }
  ]
}

Lambda

We need to create 2 lambda functions this, and if you really want you can rework both functions into one.

So for both lambda functions you create a new Python Lambda function, with the respective code and existing role lambda_start_stop_ec2, and we have our functions that can start and stop our instances

Start Instance

This function is responsible for starting the instances.

import boto3

def lambda_handler(event, context):

    if not bool(event):
        raise Exception('No event defined!')
        
    instances = event.get("instances")
    region = event.get("region")
        
    if not bool(instances):
        raise Exception('Instances are not defined!')
        
    if not type(instances) is list:
        raise Exception('Instances is not a list!')
        
    if not bool(instances):
        raise Exception('No instances defined')

    if not bool(region):
        raise Exception('Region is not defined!')

    ec2 = boto3.client('ec2', region_name=region)
    ec2.start_instances(InstanceIds=instances)
        
    for instance in instances:
         print "Started {} instance in {} region".format(instance, region)

Stop Instance

This function is responsible for stopping the instances

import boto3

def lambda_handler(event, context):

    if not bool(event):
        raise Exception('No event defined!')
        
    instances = event.get("instances")
    region = event.get("region")       
    
    if not bool(instances):
        raise Exception('Instances are not defined!')
        
    if not type(instances) is list:
        raise Exception('Instances is not a list!')
        
    if not bool(instances):
        raise Exception('No instances defined')

    if not bool(region):
        raise Exception('Region is not defined!')

    ec2 = boto3.client('ec2', region_name=region)
    ec2.stop_instances(InstanceIds=instances)
        
    for instance in instances:
         print "Started {} instance in {} region".format(instance, region)

CloudWatch

Create both rules for starting and stopping the instances you have defined.

When creating new rule, set Event selector to Schedule, and select how and when to trigger it. You can add as many targets as you want to a rule, for different regions and so on.

StartInstances

New target: Lambda function
Function name: StartEC2Instances
Configure input
- Constant (JSON)

{
  "region": "us-east-1",
  "instances": [
    "i-1e91dd10"
  ]
}

StopInstances

New target: Lambda function
Function name: StopEC2Instances
Configure input
- Constant (JSON)

{
  "region": "us-east-1",
  "instances": [
    "i-1e91dd10",
    "i-1e95235f",
    "i-1e92235f"    
  ]
}

Logs

Now in the logs you can see when the instances have been started/stopped

/aws/lambda/StartEC2Instances

START RequestId: b1b0176b-ae46-11e6-b8f8-c9b72337b2e1 Version: $LATEST
Started i-1e91dd10 instance in us-east-1 region
END RequestId: b1b0176b-ae46-11e6-b8f8-c9b72337b2e1
REPORT RequestId: b1b0176b-ae46-11e6-b8f8-c9b72337b2e1	Duration: 0.27 ms	Billed Duration: 100 ms Memory Size: 128 MB	Max Memory Used: 18 MB

/aws/lambda/StopEC2Instances

START RequestId: da073d48-ae46-11e6-a365-eb4b71a100af Version: $LATEST
Stopped i-1e91dd10 instance in us-east-1 region
Stopped i-1e95235f instance in us-east-1 region
Stopped i-1e92235f instance in us-east-1 region
END RequestId: da073d48-ae46-11e6-a365-eb4b71a100af
REPORT RequestId: da073d48-ae46-11e6-a365-eb4b71a100af	Duration: 0.32 ms	Billed Duration: 100 ms Memory Size: 128 MB	Max Memory Used: 14 MB

And there we go, we have a working system that can start/instances whenever we want.

Compressing multiple files or directories with XZ and TAR

Mon, 17 Oct 2016 22:35:00 +0000

XZ is one of the best compression tools I’ve seen, it’s compressed files so big to a fraction of their size. I’ve had an 16 GB SQL dump, and it managed to compress it down to 263 MB.

I’ve needed to compress several files using XZ. And as XZ compresses single files we are gonna have to use TAR to do it

And since the newer version of XZ supports multi threading, we can speed up compression quite a bit

Since the XZ utils in Debian/Ubuntu is an old version I’ve created my own backports that I can use.

echo "deb http://packages.matoski.com/ debian main" | sudo tee /etc/apt/sources.list.d/packages-matoski-com.list
curl -s http://packages.matoski.com/keyring.gpg | sudo apt-key add -
sudo apt-get update
sudo apt-get install xz-utils

This should give you a newer version of XZ that supports multi threading.

Let’s have a directory with a lot of files and subdirectories, to compress it all we would do

tar -cf - directory/ | xz -9e --threads=8 -c - > tarfile.tar.xz

On the newer versions of tar you can also do the following too

XZ_OPT="-9e --threads=8" tar cJf tarfile.tar.xz directory

Or if you want you can do it separately with a progress bar like this.

tar -cf - directory/ | (pv -s $(du -sb directory/ | awk '{print $1}') -p --timer --rate --bytes > tarfile.tar)
xz -9ev --threads=8 tarfile.tar

This will give you a nice progress bar while it’s creating the archive firstly and then compressing it with XZ

Tracking versions in Go

Fri, 14 Oct 2016 12:23:00 +0000

Some of my services were written in Go, and we wanted a way to track which version was used, so we can track and solve issues. Or even see when the application was deployed or from which source was built. The code is on git, so adding a git commit hash to the mix will make our problem easier. Luckily go provides us with a simple way of doing this.

Lets look at some of the stuff I keep track of so I can easily see which version I’m using. I usually call this file version.go and drop it in the root of the package.

package main

import (
	"flag"
	"fmt"
	"os"
	"runtime"
)

var version = flag.Bool("version", false, "Print the version and exit")

var BuildVersion string
var BuildHash string
var BuildDate string
var BuildClean string
var Name = "application-name"

func init() {
	flag.Parse()
	if *version {
		fmt.Printf("Name: %s\n", Name);
		fmt.Printf("Version: %s\n", BuildVersion);
		fmt.Printf("Git Commit Hash: %s\n", BuildHash);
		fmt.Printf("Build Date: %s\n", BuildDate);
		fmt.Printf("Built from clean source tree: %s\n", BuildClean)
		fmt.Printf("OS: %s\n", runtime.GOOS)
		fmt.Printf("Architecture: %s\n", runtime.GOARCH)
		os.Exit(1)
	}
}

Lets also create a file called VERSION, and we use that file to keep track of what version is our application. My version file contains the following information

$ cat VERSION
0.0.1

To build the application that will show me the version details I would issue the following command, I usually write a Makefile that will generate the command bellow, with all the necessary information.

go build -ldflags "-X 'main.BuildVersion=$(cat VERSION)' -X 'main.BuildHash=$(git log --pretty=format:'%H' -n 1)' -X 'main.BuildDate=$(date -u)' -X 'main.BuildClean=no'" -o app .

After building the application, we can run it and see the following details

$ ./app -version
Name: application-name
Version: 0.0.1
Git Commit Hash: 5f80f2f91251474da7e92ccddbcabecad0fea35d
Build Date: Thu Oct 13 22:16:08 UTC 2016
Built from clean source tree: no
OS: linux
Architecture: amd64

Usually all our errors are logged to an elasticsearch cluster, and we can easily track and check if there are any errors and from which services the errors occurred.

And here is the Makefile that I use to automate this

.DEFAULT_GOAL := build

BIN_DIR=bin
BIN_FILE=application-name
VERSION_FILE=VERSION
GO_BIN=$(shell which go)

HAS_GO_BIN := $(shell command -v go 2> /dev/null)

GIT_STATUS=$(shell git status --porcelain)
BUILD_VERSION:=$(shell git log --pretty=format:'%h' -n 1)
BUILD_DATE:=$(shell date -u)

ifneq ($(wildcard $(VERSION_FILE)),)
	VERSION:=$(shell cat $(VERSION_FILE))
else
	VERSION:=
endif

ifeq ($(GIT_STATUS),)
  BUILD_CLEAN=yes
else
  BUILD_CLEAN=no
endif

.PHONY: check-env
check-env:
ifndef GOPATH
	$(error GOPATH is undefined, you need to configure your system ...)
endif
ifndef GOROOT
	$(error GOROOT is undefined, you need to configure your system ...)
endif
ifndef HAS_GO_BIN
	$(error go command couldn't be found in PATH)
endif

bin:
	mkdir ${BIN_DIR}

.PHONY: build
build: ${BIN_DIR} check-env
	${GO_BIN} build -ldflags "-X 'main.BuildVersion=${VERSION}' -X 'main.BuildHash=${BUILD_VERSION}' -X 'main.BuildDate=${BUILD_DATE}' -X 'main.BuildClean=${BUILD_CLEAN}'" -o "${BIN_DIR}/${BIN_FILE}" .

.PHONY: clean
clean:
	rm -fv ${BIN_DIR}/${BIN_FILE}

Let me know in the comments if you use some other ways of versioning your go application.

Dropping all databases in a MongoDB

Wed, 28 Sep 2016 12:32:00 +0000

I needed to drop all the databases or based on a filter in MongoDB.

The easiest way for would be to run the following javascript command.

var dbs = db.getMongo().getDBNames();
dbs.forEach(function(dbName) {
    dbd = db.getMongo().getDB( dbName );
        print("Dropping " + dbName + " database");
        db.dropDatabase();
})

If you want to actually filter on this you can always add a condition to the drop, like drop all the databases that begin with stats_ you can do like this

var dbs = db.getMongo().getDBNames();
dbs.forEach(function(dbName) {
    dbd = db.getMongo().getDB( dbName );
    if (dbName.startsWith("stats_")) {
        print("Dropping " + dbName + " database");
        db.dropDatabase();
    }
})

Dropping all collections in a MongoDB database

Sat, 13 Aug 2016 14:23:00 +0000

I needed to drop all the collections in a database without dropping the database.

The easiest way for would be to run the following javascript command.

var dbName = 'exampleDb';
db.getSiblingDB(dbName).getCollectionNames().forEach(function(collName) {
    if (!collName.startsWith("system.")) {
        print("Dropping ["+dbName+"."+collName+"]");
        db[collName].drop();
    }
})

Interview Question - Fibonachos User Story

Fri, 19 Feb 2016 12:00:00 +0000

As an avid foodie, I want to eat Fibonachos. I want to see a program that will print the Fibonacci sequence described below so that I can assess how many I can fit on a plate, and assess basic to moderate programming skills in a language of the candidate’s choice.

Fibonacci numbers or Fibonacci sequence are the numbers in the following integer sequence 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987

The program should print the first 30 Fibonacci numbers where the definition of a Fibonacci number is

f(0) = 0
f(1) = 1
f(n) = f(n-1) + f(n-2)
n > 0

Calculating Fibonacci is quite easy as the formula is quite simple, there are several ways to implement the calculations, Recursive algorithm (extremely slow), Dynamic programming (slow), Matrix exponentiation, Fast doubling.

One thing to note is that because JavaScript uses floating points for integers, after n=78, the results are not correct.

For example Fibonacci(80) = 23416728348467685 but we get 23416728348467684.

Recursive algorithm (extremely slow)

We can directly execute the recurrence as given in the mathematical definition of the Fibonacci sequence. But it’s extremely slow: It uses O(n) stack space and O(?n)) arithmetic operations, where ?=5?+12?=5+12 (the golden ratio). In other words, the number of operations to compute F(n) is proportional to the resulting value itself, which grows exponentially.

function fibonacci(n) {
    if (n < 0 || isNaN(parseInt(n))) {
        throw new Error("Invalid number or less than zero");
    }
    if (n === 0 || n === 1) {
        return n;
    }
    return fibonacci(n - 1) + fibonacci(n - 2);
}

Iterative (slow)

It should be clear that if we’ve already computed F(n-2) and F(n-1), then we can add them to get F(n). We repeat until we reach i=n.

function fibonacci(n) {
    if (n < 0 || isNaN(parseInt(n))) {
        throw new Error("Invalid number or less than zero");
    }
    if (n === 0 || n === 1) {
        return n;
    }
    var seq = [0, 1];
    for (i = 2; i <= n; i++) {
        seq[i] = seq[i - 1] + seq[i - 2];
    }
    return seq[n];
}

With memoization (Dynamic Programming)

This is quite simple as you can see, but using it on bigger numbers will be slower on repetitive calculations, one thing we can do is do memoization of the number, so the next calculations are faster.

function Fibonacci() {
    return {
        dictionary: [0, 1],
        recursive: function(n) {
            if ( n < 0 || isNaN(parseInt(n)) ) {
                throw new Error("Invalid number or less than zero");
            }
            if (n === 0 || n === 1) {
                return n;
            }
            if (this.dictionary[n]) {
                return this.dictionary[n];
            }
            this.dictionary[n] = this.recursive(n - 1) + this.recursive(n - 2);
            return this.dictionary[n];
        },
        iterative: function(n) {
            if ( n < 0 || isNaN(parseInt(n)) ) {
                throw new Error("Invalid number or less than zero");
            }
            if (n === 0 || n === 1) {
                return n;
            }
            if (this.dictionary[n]) {
                return this.dictionary[n];
            }
            for (i = 2; i <= n; i++) {
                this.dictionary[i] = this.dictionary[i - 1] + this.dictionary[i - 2];
            }
            return this.dictionary[n];
        }
    };
}

Fast doubling method

The fast doubling method is even better, running in O(log(n)) time.

It’s defined like

F(2n)   = F(n) * (2*F(n+1) - F(n))
F(2n+1) = F(n+1)^2 + F(n)^2

And here is a sample implementation of the code

function fibonacci(n) {

    if ( n < 0 || isNaN(parseInt(n)) ) {
        throw new Error("Invalid number or less than zero");
    }

    if ( n == 0 ){
        return 0;
    }
    
    return fibonacci_tuple(n-1)[1];
}

function fibonacci_tuple(n) {

    if ( n < 0 || isNaN(parseInt(n)) ) {
        throw new Error("Invalid number or less than zero");
    }

    if (n == 0)
        return [0, 1];
    
    var result = fibonacci_tuple(Math.floor(n/2));
    var a = result[0], b = result[1];
    
    c = a * ( b * 2 - a );
    d = a * a + b * b;
    if ( n % 2 == 0 ) {
        return [c, d];
    } else {
        return [d, c + d];
    }
}

Matrix exponentiation

[1 1]^n   [F(n+1) F(n)  ]
[1 0]   = [F(n)   F(n-1)]

Verifying SSH Key Fingerprints with DNS records

Sat, 30 Jan 2016 11:48:46 +0100

SSH Fingerprinting is a method to provide DNS records for key fingerprint verification of any client that logs into said machine.

Doing this will prevent users from blindly typing ‘yes’ when asked if they want to continue connecting to an SSH host who’s authenticity is unknown.

Most of the people just type ‘yes’ without even checking if it’s correct or not, which defeats the purpose of the prompt.

The fingerprint records together with DNSSEC will completely bypass the prompt and have SSH verify the fingerprint automatically.

If the authenticity of the host is unknown, you haven’t been logged in before or you have changed the fingerprint of the machine, you will be greeted by a very familiar prompt, and most users here will just type yes without even checking if really we are connecting to the correct machine.

$ ssh root@myserverplace.de
The authenticity of host 'myserverplace.de (148.251.100.157)' can't be established.
ECDSA key fingerprint is SHA256:y2STsQ4RA/8durhpic+pb6UjcKwz7+bUaKX3C40yOGk.
Are you sure you want to continue connecting (yes/no)?
Warning: Permanently added 'myserverplace.de' (ECDSA) to the list of known hosts.

Generating SSH Key Fingerpint records

If you want to generate the DNS records you need to login to the said server and run ssh-keygen to generate the records, and then add the relevant records to your DNS server.

$ ssh-keygen -r myserverplace.de
myserverplace.de IN SSHFP 1 1 db744817e8d6ac2027e6629aac7f0fc1750f6588
myserverplace.de IN SSHFP 1 2 a61db02b9b26ca48663c3272821b451773c7cd1e9a412f5a09994ec8f8738c79
myserverplace.de IN SSHFP 2 1 493a9e6a4b5078b1d0c5424aecf817ea54e1dfdf
myserverplace.de IN SSHFP 2 2 064b9dd10805069eb508bd087a37db61fda2107138924112ded3ccdbaafd6cb3
myserverplace.de IN SSHFP 3 1 7c4b9b9105d6a0d7aacf44534a7800fc10466683
myserverplace.de IN SSHFP 3 2 cb6493b10e1103ff1dbab86989cfa96fa52370ac33efe6d468a5f70b8d323869
myserverplace.de IN SSHFP 4 1 69ac080ccf6cd52f4788373bd4dca21731e69713
myserverplace.de IN SSHFP 4 2 7cae4ff942899f8e155bfc675e72e4146a1bf4107977fe73c6cffa8f3fda8fc3

Each line contains the following information

#	Hostname	Algorithm	Fingerprint Type	Hash
1	myserverplace.de	RSA	SHA-1	db744817e8d6ac2027e6629aac7f0fc1750f6588
2	myserverplace.de	RSA	SHA-2	a61db02b9b26ca48663c3272821b451773c7cd1e9a412f5a09994ec8f8738c79
3	myserverplace.de	DSA	SHA-1	493a9e6a4b5078b1d0c5424aecf817ea54e1dfdf
4	myserverplace.de	DSA	SHA-2	064b9dd10805069eb508bd087a37db61fda2107138924112ded3ccdbaafd6cb3
5	myserverplace.de	ECDSA	SHA-1	7c4b9b9105d6a0d7aacf44534a7800fc10466683
6	myserverplace.de	ECDSA	SHA-2	cb6493b10e1103ff1dbab86989cfa96fa52370ac33efe6d468a5f70b8d323869
7	myserverplace.de	ED25519	SHA-1	69ac080ccf6cd52f4788373bd4dca21731e69713
8	myserverplace.de	ED25519	SHA-2	7cae4ff942899f8e155bfc675e72e4146a1bf4107977fe73c6cffa8f3fda8fc3

Algorithm

RSA
DSA
ECDSA
ED25519

You should never use DSA or ECDSA, Ed25519 is probably the strongest mathematically (and also the fastest), but not yet widely supported. As a bonus, it has stronger encryption (password-protection) of the private key by default than other key types. RSA is the best bet if you can’t use Ed25519.

Fingerprint type

SHA-1
SHA-2

You shouldn’t use SHA-1 fingerprints as they are less secure.

DNS Records

After adding the DNS records you can check if they are present or not very simple

$ dig SSHFP +noadditional +noquestion +nocomments +nocmd +nostats myserverplace.de
myserverplace.de.	10539	IN	SSHFP	4 2 7CAE4FF942899F8E155BFC675E72E4146A1BF4107977FE73C6CFFA8F 3FDA8FC3
myserverplace.de.	10539	IN	SSHFP	3 1 7C4B9B9105D6A0D7AACF44534A7800FC10466683
myserverplace.de.	10539	IN	SSHFP	3 2 CB6493B10E1103FF1DBAB86989CFA96FA52370AC33EFE6D468A5F70B 8D323869
myserverplace.de.	10539	IN	SSHFP	4 1 69AC080CCF6CD52F4788373BD4DCA21731E69713

SSH Client

You need to add VerifyHostKeyDNS to your ~/.ssh/config or your global /etc/ssh/ssh_config, if you set this property then SSH will try to validate host keys from DNS.

Host *
    VerifyHostKeyDNS yes

Connecting to SSH with SSHFP

Now if I connect to the server I get the following prompt, without DNSSEC you will still get a prompt, and Matching host key fingerprint found in DNS line in the output now.

$ ssh root@myserverplace.de
The authenticity of host 'myserverplace.de (148.251.100.157)' can't be established.
ECDSA key fingerprint is SHA256:y2STsQ4RA/8durhpic+pb6UjcKwz7+bUaKX3C40yOGk.
Matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)?

With DNSSEC in place it will not add a new entry in ~/.ssh/known_hosts, and will connect right away.

References

Automatically switch between WiFi and Ethernet

Sun, 20 Dec 2015 22:25:00 +0000

I wanted a way to automatically switch between wifi and ethernet when I connect the system to the ethernet port, or auto start wifi when I disconnect the ethernet port.

Simplest way to do so was to use NetworkManager to enable/disable wifi, when we detect an ethernet connection.

I also wanted a way to track what is happening so we log to syslog our actions, and a way to disable the functionality without removing the file from they system.

Open up your favorite editor and create the file bellow with the contents.

/etc/NetworkManager/dispatcher.d/70-wifi-wired-exclusive.sh

#!/usr/bin/env bash

name_tag="wifi-wired-exclusive"
syslog_tag="$name_tag"
skip_filename="/etc/NetworkManager/.$name_tag"

if [ -f "$skip_filename" ]; then
  exit 0
fi

interface="$1"
iface_mode="$2"
iface_type=$(nmcli dev | grep "$interface" | tr -s ' ' | cut -d' ' -f2)
iface_state=$(nmcli dev | grep "$interface" | tr -s ' ' | cut -d' ' -f3)

logger -i -t "$syslog_tag" "Interface: $interface = $iface_state ($iface_type) is $iface_mode"

enable_wifi() {
   logger -i -t "$syslog_tag" "Interface $interface ($iface_type) is down, enabling wifi ..."
   nmcli radio wifi on
}

disable_wifi() {
   logger -i -t "$syslog_tag" "Disabling wifi, ethernet connection detected."
   nmcli radio wifi off
}

if [ "$iface_type" = "ethernet" ] && [ "$iface_mode" = "down" ]; then
  enable_wifi
elif [ "$iface_type" = "ethernet" ] && [ "$iface_mode" = "up"  ] && [ "$iface_state" = "connected" ]; then
  disable_wifi
fi

If we want to disable this script, instead of deleting it we can just do

touch /etc/NetworkManager/.wifi-wired-exclusive

And this will skip the script everytime it’s invoked.

And here are some details from the syslog, as you can see it turns off the wifi if there is an ethernet connection on the system, and turn on the wifi if the ethernet connection is disabled.

Dec 20 22:47:13 x1 wifi-wired-exclusive[7980]: Interface: enx0050b6c0408e = connected (ethernet) is up
Dec 20 22:47:13 x1 wifi-wired-exclusive[7981]: Disabling wifi, ethernet connection detected.
Dec 20 22:47:13 x1 wifi-wired-exclusive[8051]: Interface: wlp4s0 = unavailable (wifi) is down
Dec 20 22:47:21 x1 wifi-wired-exclusive[8097]: Interface: enx0050b6c0408e = unavailable (ethernet) is down
Dec 20 22:47:21 x1 wifi-wired-exclusive[8098]: Interface enx0050b6c0408e (ethernet) is down, enabling wifi ...
Dec 20 22:47:25 x1 wifi-wired-exclusive[8252]: Interface: wlp4s0 = connected (wifi) is up
Dec 20 22:47:31 x1 wifi-wired-exclusive[8407]: Interface: enx0050b6c0408e = connected (ethernet) is up
Dec 20 22:47:31 x1 wifi-wired-exclusive[8408]: Disabling wifi, ethernet connection detected.
Dec 20 22:47:31 x1 wifi-wired-exclusive[8474]: Interface: wlp4s0 = unavailable (wifi) is down
Dec 20 22:47:42 x1 wifi-wired-exclusive[8525]: Interface: enx0050b6c0408e = unavailable (ethernet) is down
Dec 20 22:47:42 x1 wifi-wired-exclusive[8526]: Interface enx0050b6c0408e (ethernet) is down, enabling wifi ...
Dec 20 22:47:46 x1 wifi-wired-exclusive[8679]: Interface: wlp4s0 = connected (wifi) is up

Debian daily backup domains with tartarus.

Sun, 14 Jun 2015 10:13:58 +0000

In a previous article we saw how to set up a backup process for tartarus, in this article we will see how we can generate backups of the data in each vhost with tartarus, the reason I do this is so we can set up a daily incremental backup of the files for each domain.

Make sure you read the article Debian Backup with Tartarus, this will prepare you for the basics of this article.

Now lets create some files that we can use for this:

/etc/tartarus/generic.inc

STORAGE_METHOD="FILE"
STORAGE_SERVER="example.com"
STORAGE_USER="user"
STORAGE_DIR="backup"
STORAGE_FILE_DIR="/mnt/$STORAGE_SERVER/$STORAGE_DIR/"
WEEK_NUMBER=`date +%V`

LIMIT_DISK_IO="yes"

FILE_LIST_CREATION="yes"
FILE_LIST_DIRECTORY="/mnt/$STORAGE_SERVER/$STORAGE_DIR/"

COMPRESSION_METHOD="bzip2"

STAY_IN_FILESYSTEM="yes"

TAR_OPTIONS="--preserve-permissions"

ENCRYPT_SYMMETRICALLY="no"
ENCRYPT_PASSPHRASE_FILE="/etc/tartarus/backup.sec"

TARTARUS_PRE_PROCESS_HOOK() {

  if [ ! -d "/mnt/$STORAGE_SERVER" ]; then
    mkdir -p "/mnt/$STORAGE_SERVER"
  fi
  
  umount "/mnt/$STORAGE_SERVER"
  sshfs $STORAGE_USER@$STORAGE_SERVER:/ /mnt/$STORAGE_SERVER -o reconnect

  if [ ! -d "$STORAGE_FILE_DIR" ]; then
     mkdir -p "$STORAGE_FILE_DIR"
  fi

  if [ ! -d "$FILE_LIST_DIRECTORY" ]; then
     mkdir -p "$FILE_LIST_DIRECTORY"
  fi

}

TARTARUS_PRE_CLEANUP_HOOK() {
  echo "Cleaning ..." | logger -i
}

TARTARUS_POST_CLEANUP_HOOK() {
   umount "/mnt/$STORAGE_SERVER"
}

TARTARUS_DEBUG_HOOK() {
    echo $DEBUGMSG | logger -i
}

TARTARUS_PRE_FREEZE_HOOK() {
  sync
}

We need to supply the proper information for STORAGE_USER and STORAGE_SERVER in the bash script above.

/etc/tartarus/vhost.template

source /etc/tartarus/generic.inc
NAME="%NAME%"
DIRECTORY="%DIRECTORY%"
EXCLUDE=""
CREATE_LVM_SNAPSHOT="no"
INCREMENTAL_STACKING="yes"
INCREMENTAL_TIMESTAMP_FILE="/var/spool/tartarus/timestamps/%NAME%"
STORAGE_FILE_DIR="$STORAGE_FILE_DIR/vhosts/%NAME%/archives/${WEEK_NUMBER}"
FILE_LIST_DIRECTORY="$FILE_LIST_DIRECTORY/vhosts/%NAME%/lists"

/etc/tartarus/create.sh

#!/bin/bash
items=`find /var/www/vhosts/ -maxdepth 1 -group psaserv`

for itm in $items; do
  basefile=$(basename $itm)
  name=$basefile
  dir=$itm
  ./create_vhost.sh "$name" "$dir"
done

/etc/tartarus/create_vhost.sh

#!/bin/bash

if [ "$#" -ne 2 ]
then
  echo "Usage: ./create_vhost.sh <name> <directory>"
  exit 1
fi

NAME=$1
DIRECTORY=$2

sed -e "s:%DIRECTORY%:$DIRECTORY:g" -e "s:%NAME%:$NAME:g" vhost.template > "vhost_${NAME}.conf"

And there you go, now calling the script /etc/tartarus/create.sh will create all the required files for the hosts, you will need to re-run this whenever you add new domains, so you can also run this periodically, once a week, for the first full backup.

Touchpad not working on Debian after suspend/hibernate

Fri, 12 Jun 2015 01:00:44 +0000

Touchpad doesn’t work after suspend/hibernate on Debian, I ran into this issue a while ago when I installed Debian on my MSI GT70 DRAGON EDITION laptop, this probably will apply for other laptops too.

Turns out the solution is simple, you just need to reinitialize the touchpad, can be done by simple commands.

modprobe -r psmouse
modprobe psmouse

Now it will work, but let’s see if we can make it automated.

So firstly let’s install some prerequisites before continuing.

apt-get install pm-utils

After installing you should create the following file.

/etc/pm/sleep.d/00_trackpad

#!/bin/sh
case "$1" in
    suspend|hibernate)
         modprobe -r psmouse ;;
    resume|thaw)
        modprobe psmouse ;;
        synclient TouchpadOff=0 ;;
esac

Now suspend and hibernate will work properly when you resume the system.

Some DMI information about my system

$ grep '.*' /sys/class/dmi/id/*_* 2>/dev/null
/sys/class/dmi/id/bios_date:11/08/2012
/sys/class/dmi/id/bios_vendor:American Megatrends Inc.
/sys/class/dmi/id/bios_version:E1762IMS.70T
/sys/class/dmi/id/board_asset_tag:To be filled by O.E.M.
/sys/class/dmi/id/board_name:MS-1762
/sys/class/dmi/id/board_serial:BSS-0123456789
/sys/class/dmi/id/board_vendor:Micro-Star International Co., Ltd.
/sys/class/dmi/id/board_version:REV:1.0
/sys/class/dmi/id/chassis_asset_tag:No Asset Tag
/sys/class/dmi/id/chassis_serial:None
/sys/class/dmi/id/chassis_type:10
/sys/class/dmi/id/chassis_vendor:Micro-Star International
/sys/class/dmi/id/chassis_version:N/A
/sys/class/dmi/id/product_name:GT70
/sys/class/dmi/id/product_serial:FFFFFFFF
/sys/class/dmi/id/product_uuid:00000000-0000-0000-0000-8C89A50783FA
/sys/class/dmi/id/product_version:REV:1.0
/sys/class/dmi/id/sys_vendor:Micro-Star International Co., Ltd.

Coding Problem - Prime Path

Tue, 02 Jun 2015 23:24:00 +0000

The ministers of the cabinet were quite upset by the message from the Chief of Security stating that they would all have to change the four-digit room numbers on their offices. * It is a matter of security to change such things every now and then, to keep the enemy in the dark. * But look, I have chosen my number 1033 for good reasons. I am the Prime minister, you know! * I know, so therefore your new number 8179 is also a prime. You will just have to paste four new digits over the four old ones on your office door. * No, it’s not that simple. Suppose that I change the first digit to an 8, then the number will read 8033 which is not a prime! * I see, being the prime minister you cannot stand having a non-prime number on your door even for a few seconds. * Correct! So I must invent a scheme for going from 1033 to 8179 by a path of prime numbers where only one digit is changed from one prime to the next prime.

Now, the minister of finance, who had been eavesdropping, intervened.

No unnecessary expenditure, please! I happen to know that the price of a digit is one pound.
Hmm, in that case I need a computer program to minimize the cost. You don’t know some very cheap software gurus, do you?
In fact, I do. You see, there is this programming contest going on…

Help the prime minister to find the cheapest prime path between any two given four-digit primes! The first digit must be nonzero, of course. Here is a solution in the case above.

The cost of this solution is 6 pounds. Note that the digit 1 which got pasted over in step 2 can not be reused in the last step – a new 1 must be purchased.

Input

One line with a positive number: the number of test cases (at most 100). Then for each test case, one line with two numbers separated by a blank. Both numbers are four-digit primes (without leading zeros).

Output

One line for each case, either with a number stating the minimal cost or containing the word “Impossible”.

6
7
0

Solution

Lets see how would we do the algorithm for this, first before we proceed we need a way to check if the number is prime or not, so we will need a list of prime numbers up to 10000.

To do this we can use Sieve of Eratosthenes, Trial division, or some other method to generate a list of prime numbers.

This is a graph problem, so we can use Breadth-first search or Dijkstra’s algorithm algorithms to find the shortest path between the number that we want.

Now let’s see a working implementation of this in Java

Java

import java.io.BufferedInputStream;
import java.util.LinkedList;
import java.util.Scanner;

class PrimePath {

    static boolean[] isPrime = getPrimes(10000);

    public static boolean[] getPrimes(int n) {
        int i, j, k, x;
        boolean[] a = new boolean[n];
        n++;
        n /= 2;
        int[] b = new int[(n + 1) * 2];
        a[2] = true;
        a[3] = true;
        for (i = 1; i <= 2 * n; i++) {
            b[i] = 0;
        }
        for (i = 3; i <= n; i += 3) {
            for (j = 0; j < 2; j++) {
                x = 2 * (i + j) - 1;
                while (b[x] == 0) {
                    a[x] = true;
                    for (k = x; k <= 2 * n; k += x) {
                        b[k] = 1;
                    }
                }
            }
        }
        return a;
    }

    public static void main(String[] args) {

        Scanner scan = new Scanner(new BufferedInputStream(System.in));
        int cas = scan.nextInt();
        for (int i = 1; i <= cas; i++) {
            int start = scan.nextInt();
            int end = scan.nextInt();
            boolean[] isVisited = new boolean[10000];
            int[] step = new int[10000];
            LinkedList<Integer> queue = new LinkedList();
            queue.addLast(start);
            isVisited[start] = true;
            while (!queue.isEmpty()) {
                int current = queue.pop();
                if (current == end) {
                    break;
                }
                for (int j = 0; j <= 9; j++) {
                    int next1 = getNext(1, j, current);
                    int next2 = getNext(2, j, current);
                    int next3 = getNext(3, j, current);
                    int next4 = getNext(4, j, current);
                    if (!isVisited[next1]) {
                        queue.addLast(next1);
                        step[next1] = step[current] + 1;
                        isVisited[next1] = true;
                    }
                    if (!isVisited[next2]) {
                        queue.addLast(next2);
                        step[next2] = step[current] + 1;
                        isVisited[next2] = true;
                    }
                    if (!isVisited[next3]) {
                        queue.addLast(next3);
                        step[next3] = step[current] + 1;
                        isVisited[next3] = true;
                    }
                    if (!isVisited[next4]) {
                        queue.addLast(next4);
                        step[next4] = step[current] + 1;
                        isVisited[next4] = true;
                    }
                }
            }
            System.out.println(step[end]);
        }
    }

    public static int getNext(int flag, int i, int current) {
        int next = 0;
        if (flag == 1) {
            if (i == 0) {
                return current;
            }
            next = current % 1000 + i * 1000;
        }
        if (flag == 2) {
            int t = current / 1000;
            next = t * 1000 + current % 1000 % 100 + i * 100;
        }
        if (flag == 3) {
            int t = current / 100;
            int tt = current % 10;
            next = t * 100 + i * 10 + tt;
        }
        if (flag == 4) {
            next = current / 10 * 10 + i;

        }
        if (!isPrime[next]) {
            return current;
        }
        return next;
    }

}

Coding Problem - Phone List

Tue, 02 Jun 2015 16:03:00 +0000

Given a list of phone numbers, determine if it is consistent in the sense that no number is the prefix of another. Let’s say the phone catalogue listed these numbers:

Emergency 911
Alice 97 625 999
Bob 91 12 54 26

In this case, it’s not possible to call Bob, because the central would direct your call to the emergency line as soon as you had dialled the first three digits of Bob’s phone number. So this list would not be consistent.

Input

The first line of input gives a single integer, 1≤t≤40, the number of test cases. Each test case starts with n, the number of phone numbers, on a separate line, 1≤n≤10000. Then follows n lines with one unique phone number on each line. A phone number is a sequence of at most ten digits.

Output

For each test case, output “YES” if the list is consistent, or “NO” otherwise.

NO
YES

Solution

Lets see how we would go about solving this.

Since the maximum length of a number is 10 digits, we would iterate over the list backwards, and check for doubles.

To be able to detect if the list is consistent or not, the algorithm would be something along these lines.

Algorithm 1

Group the numbers by length in an array (Hash tree)
Check for doubles, if there is a double then it’s not consistent, don’t proceed with the test, even though it says it’s unique, you never know what could happen, so it doesn’t hurt to try, and it doesn’t hurt the performance too much
Iterate over the the array backwards from the biggest available number length
1. Check for doubles, and if there are it’s not consistent, so break
2. If MAX_NUMBER_LENGTH = CURRENT_NUMBER_LENGTH or CURRENT_NUMBER_LENGTH == MAX_NUMBER_LENGTH_IN_LIST, then we continue as it’s consistent and there are no duplicates
3. Build the array from the previous data set and merge into current so we can keep the results
4. Create an array the we gonna use to compare
5. Check if the keys are present in the array
6. Break if we detected a match
7. Add the values to current index, so we can compare with previous one on the next run

This is implemented in PHP

Algorithm 2

Another way to do this is to use a Tree, and each of the Nodes needs to have 10 leafs (one for each digit of a number), and you build the Tree by appending another Tree onto the leaf. If it’s a new leaf (the node is consistent if the digit position of the processing number equals the length of the number). If it’s not a new leaf, but the node is consistent, then the list of number is not consistent, also the same applies if the digit position of the processing number equals the length of the number.

This is implemented in Java

Implementations

There are a lot of ways to do this, some faster some slower, a slower one would be to use a sort algorithm, but this is not good for really big values.

The input sample provided by the problem is OK, but we need better one, so here is firstly a script that will generate random test cases, for us to test. Simplest is here in PHP, but we can use it for testing in the other languages too.

<?php
$test = rand(1, 40);
$testname = utime();
file_put_contents("rt-$testname", "$test\n");

$line = 1;

for ($i = 0; $i < $test; $i++) {
    $numbers = rand(1, 10000);
    file_put_contents("rt-$testname", "$numbers\n", FILE_APPEND);
    $line++;
    for ($n = 0; $n < $numbers; $n++) {
        $number = rand(100, 9999999999);
        file_put_contents("rt-$testname", "$number\n", FILE_APPEND);
        $line++;
    }
}

PHP

<?php

$output = array();

$stdin = fopen('php://stdin', 'r');
$totalTests = trim(fgets($stdin));
$maxLengthNumber = 10;

for ($i = 0; $i < $totalTests; $i++) {

    $totalNumbers = trim(fgets($stdin));

    $list = array();
    $numbers = array();

    // 1. Group the numbers by length in an array
    for ($a = 0; $a < $totalNumbers; $a++) {
        $phone = trim(fgets($stdin));
        $numbers[] = $phone;
        $digits = strlen($phone);
        $list[$digits][] = $phone;
    }

    if ($totalNumbers != count(array_flip($numbers))) {
        // 2. Check for doubles, if there is a double then it's not consistent
        $isConsistent = false;
    } else {
        $isConsistent = true;
        // 3. Iterate over the the array
        $keys = array_keys($list);
        sort($keys);
        $keys = array_reverse($keys, SORT_NUMERIC);
        $max_key = max($keys);
        foreach ($keys as $index => $key) {

            $totalItems = count($list[$key]);

            // 3.1. Check for doubles, and if there are it's not consistent, so break
            if ($totalItems != count(array_flip($list[$key]))) {
                $isConsistent = false;
                break;
            }

            // 3.2. If MAX_NUMBER_LENGTH = CURRENT_NUMBER_LENGTH || $key == $max_key, then we continue as it's consistent and there are no duplicates
            if ($maxLengthNumber == $key || $key == $max_key) {
                $isConsistent = true;
                continue;
            }

            // 3.3. Build the array from the previous data set and merge into current so we can keep the results
            $prevCut = array();
            $prev = array();
            $currentListIndex = $keys[$index];
            $prevListIndex = $keys[$index + 1];

            // 3.4. Create an array the we gonna use to compare
            foreach ($list[$prevListIndex] as $idx => &$val) {
                $prevCut[$idx] = substr($val, 0, $key);
                $prev[$idx] = $val; // point it as a reference for faster memory access
            }

            // 3.5. Check if the keys are present in the array
            $prevCutKeys = array_flip($prevCut);
            foreach ($list[$currentListIndex] as $idx => $val) {

                if (isset($prevCutKeys[$val])) {
                    $isConsistent = false;
                    break;
                } else {
                    $isConsistent = true;
                }

            }

            // 3.6. Break if we detected a match
            if (!$isConsistent) {
                break;
            }

            // 3.7. Add the values to current index, so we can compare with previous one on the next run
            foreach ($prev as $val) {
                $list[$currentListIndex][] = $val;
            }

        }
    }
    array_push($output, $isConsistent ? "YES" : "NO");

}

$out = fopen('php://output', 'w'); //output handler
fputs($out, join("\n", $output) . "\n"); //writing output operation
fclose($out);

Java

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

public class PhoneList {

    public static final String YES = "YES";
    public static final String NO = "NO";
    private static Tree tree = new Tree();
    private static boolean isConsistent = true;

    private static class Tree {
        int node = 0;
        boolean isCons = false;
        Tree[] next = new Tree[10];

        public void setNext(int index, Tree t) {
            next[index] = t;
        }

        public Tree getTree(int index) {
            return next[index];
        }
    }

    // Should be on a separate class
    public static void main(String[] args) {
        PhoneList phoneList = new PhoneList();
        try {
            phoneList.build();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public void build() throws IOException {
        BufferedReader reader = new BufferedReader(new InputStreamReader(System.in));
        int t = Integer.parseInt(reader.readLine());
        for (int i = 0; i < t; ++i) {
            int n = Integer.parseInt(reader.readLine());
            tree = new Tree();
            isConsistent = true;
            for (int j = 0; j < n; ++j) {
                String phone = reader.readLine();
                if (isConsistent)
                    buildTree(phone);
            }
            if (isConsistent)
                System.out.println(YES);
            else
                System.out.println(NO);
        }
    }

    private void buildTree(String s) {
        int len = s.length();
        Tree auxTree = tree;
        for (int i = 0; i < len; ++i) {
            int ch = Integer.parseInt(s.substring(i, i + 1));
            Tree aux = auxTree.next[ch];
            if (aux == null) {
                aux = new Tree();
                aux.node = 1;
                if (i == len - 1)
                    aux.isCons = true;
                auxTree.setNext(ch, aux);
                auxTree = aux;
            } else {
                if (aux.isCons) {
                    isConsistent = false;
                    break;
                }
                if (i == len - 1) {
                    isConsistent = false;
                    break;
                }
                auxTree = aux;
            }
        }
    }

}

Amazon Glacier Interface

Wed, 14 Jan 2015 14:55:00 +0000

Amazon Glacier is an archive/backup service with very low storage price. However with some caveats in usage and archive retrieval prices. Read more about Amazon Glacier

I used a couple of utilities on linux and windows in the past for uploading data to Amazon Glacier servers, though I haven’t tested this on Windows, I don’t see any reason for this not to work on Windows.

I needed a tool that will make this easier to do so from the command line, that should work on Linux/Windows, and support some of the metadata from other tools. Also the tools I was using were good but none of them meet all my needs so I decided to write my own.

So I want to introduce a glacier interface for Linux a command line tool to interface with glacier.

It’s a multithreaded application, that supports multipart uploads to Amazon Glacier servers, you can specify concurrency, to speed up the upload.

Anyone is welcome to help in development of this tool, see a bug, or a want a new feature, send me a pull request, or create an issue and I will see what I can do.

Any contributors will be added to the list of contributors.

This tool is still in Beta, though I’ve used it for a lot of stuff and have uploaded close to 1 TB of data so far and haven’t run into an issue so far with the uploads.

Take a look at the README.md at the projects GitHub page.

There are a lot of examples, and how to use.

Installation

Debian based systems

echo "deb http://packages.matoski.com debian-gi main" | sudo tee /etc/apt/sources.list.d/debian-gi.list
sudo wget -O /etc/apt/trusted.gpg.d/pm.gpg http://packages.matoski.com/keyring.gpg
sudo apt-get update
sudo apt-get -y install glacier-interface

Source

You can checkout the code from the repository, and build it with gradle.

Source Code

Github Glacier Interface

Git delete a remote/local branch or tag

Thu, 08 Jan 2015 14:18:00 +0000

A lot of people keep asking me on Skype or chat how to delete a remote branch/tag in git, so I decided to put it in a nice little article, so I can refer it to them.

If you want to delete a local tag then you would do

git tag -d <tag name>

But if you want to delete remote tag, then the syntax is a little different

git push origin :refs/tags/<tag name>

This will delete the tag on the remote origin.

Now let’s see how to delete a local branch,

git branch -d <branch name>

And now for a remote branch,

git push origin :<branch name>

Or since version 1.7.0 now you can delete the branch like so.

git push origin --delete <branch name>

fail2ban for Wordpress and Plesk

Mon, 22 Dec 2014 22:54:00 +0000

I’m hosting quite a bit of pages that run WordPress, and one of the biggest problem that I ran into is that people don’t keep their WordPress or plugin installations updated.

I gave up on having everyone upgrade, because they all use different plugins, customized plugins, and a lot of other stuff that will break if they upgrade.

So here comes fail2ban to the rescue, we can use it to go through the logs, to configure the system to ban the users when they fail to log in, or do some bad stuff, or try to brute force their way into the wordpress installation, either through the login form or through xmlrpc.

You should install fail2ban if you haven’t already, on debian based you would just issue.

apt-get install fail2ban

If you use Plesk you can skip creation of the files, and just use the frontend to create the necessary jails and filters, also replace common.conf with apache-common.conf.

The configuration is located in /etc/fail2ban on most systems.

So firstly let’s create a filter.

In /etc/fail2ban/filter.d, create a file called wordpress-login.conf, and put the following text inside.

# Fail2Ban filter for WordPress Login
#

[INCLUDES]
before = common.conf

[Definition]
_daemon = wordpress
failregex = ^<HOST>.*].*POST.*/wp-login\.php HTTP.*
ignoreregex =

# DEV Notes:
#
# Rule Author: Ilija Matoski

What this does it creates a filter to be used in the next step. Also let’s create one filter for XMLRPC for WordPress in the same directory named wordpress-xmlrpc.conf

# Fail2Ban filter for WordPress XMLRPC
#

[INCLUDES]
before = common.conf

[Definition]
_daemon = wordpress
failregex = ^<HOST>.*].*/xmlrpc\.php.*
ignoreregex =

# DEV Notes:
#
# Rule Author: Ilija Matoski

Now let’s create two new jails, that we gonna use to ban the people who are gonna abuse this.

In the folder /etc/fail2ban/jail.d, wordpress.conf,

[wordpress-xmlrpc]

enabled  = true
filter   = wordpress-xmlrpc
action   = iptables-multiport[name=WordPressXMLRPC, port="http,https"]
logpath  = /var/log/apache2/access_log
maxretry = 10

[wordpress-login]

enabled  = true
filter   = wordpress-login
action   = iptables-multiport[name=WordPressLogin, port="http,https"]
logpath  = /var/log/apache2/access_log
maxretry = 10

If you are running Plesk, then replace the logpath with

logpath  = /var/www/vhosts/system/*/logs/*access*log
           /var/log/apache2/*access.log

So in Plesk, the action will look like.

iptables-multiport[name="<NAME_OF_FILTER>", port="http,https"]

This will read the data from all the hosts in Plesk

So what happens if someone tries to POST to wp-login.php, or hit xmlrpc.php, 10 times in a 10 minute period he will be banned for a day, also there is a jail called recidive in fail2ban, which if you get jailed 5 times with the previous filters, the IP will get ban for a week.

You can adjust the times in the configuration file, if you want to lower or increase them.

Well this should work, now let it run for a while, and you can see what does your, after 15 minutes of running this is what was present in my logs.

$ iptables -L fail2ban-WordPressLogin
Chain fail2ban-WordPressLogin (1 references)
target     prot opt source               destination
REJECT     all  --  189.1.169.141         anywhere            reject-with icmp-port-unreachable
REJECT     all  --  181.140.198.180       anywhere            reject-with icmp-port-unreachable
REJECT     all  --  117.21.191.227        anywhere            reject-with icmp-port-unreachable
RETURN     all  --  anywhere              anywhere

$ iptables -L fail2ban-WordPressXMLRPC
Chain fail2ban-WordPressXMLRPC (1 references)
target     prot opt source               destination
REJECT     all  --  117.21.191.227        anywhere            reject-with icmp-port-unreachable
RETURN     all  --  anywhere              anywhere

Interview question - About arrays

Thu, 16 Oct 2014 04:03:00 +0000

I had an interview for a job, and one question they asked was how to do something in a language I’m comfortable with.

So I told them what language would they like for me to give them a solution, I said I can do it in Java, PHP, C/C++, …, and they said that it’s not a problem, they understand everything, doesn’t mater which language it is written in, if you want to know the difference read bellow in the PHP section.

The job I was applying was for a scripting language Perl (the developers that interviewed me had experience with Perl), so in this case they said I can do it in PHP because its a scripting language too.

They wanted me to remove an item from an array, and to tell them how would I do it in the said language.

The problem is that they don’t understand the representation of array in PHP is completely different than any other language, but we will get into that in a little while.

Before we proceed lets see a little about the array data structure, and the internal workings of the array.

Array Data Structure

Arrays is a basic data structure representing a group of similar elements, accessed by an index, it can be stored effectively and accessed quite fast to the all of the elements.

An array doesn’t have an overhead per element. and any element of the array can be accessed at O(1) time by it’s index.

On the other hand, the array structure is not dynamic. Most programming languages provide a way to allocate arrays with arbitrary size (dynamically allocated arrays), the problem with that is when the space is used up, a new array will have to be generated and the old data copied into it.

The insertion and deletion of an element in an array requires to shift O(n) elements on average.

Static and dynamically allocated arrays

There are two types of arrays, and both of them are implemented differently.

Static array has a constant size and they are defined and present any time during runtime of the application.

But the dynamic array is created during runtime and may be deleted, when it’s not needed any more, and they can be quite big, even bigger than the amount of physical memory, even though they are dynamic they can’t be resized. But you can expand it by:

Creating a new array of bigger size
Copying data from old one to new one
Free the memory of the old one

Dynamic arrays

The problem when working with an array data structure is that its size cannot be changed during program run.

Internal representation

So how can we do this then ?

The application allocates memory and divides it into two parts, one part contains the data and the other one contains the free space.

In the beginning all space is free space. As you add items to the array, it will fill up, and when it has no space it will expand by creating a new larger array and copying the old contents to the new location.

storage: dynamically allocated space to store data
capacity: size of the storage
size: real size of the data

If you are worried about speed don’t be because copying the arrays is done on the hardware level and can be done very efficiently.

PHP

First of all, the arrays in PHP are not like traditional arrays, and everyone who comes to PHP or goes from PHP to other languages, are confused because arrays in PHP don’t work as expected, especially for people coming from Javascript, Java, Perl, C/C++, C#.

If you look at the syntax it is similar to other languages, so let’s take a look at a simple implementation of an array in PHP.

<?php
$element = array();
for ($i = 7; $i >= 0; --$i) {
  $element[$i] = $i;
}

print implode(' ', $element);

You would expect the following to be outputted:

0 1 2 3 4 5 6 7

But in reality, it will output

7 6 5 4 3 2 1 0

Why? Because it’s not an array, it’s an ordered map, which is something I pointed out to the interviewers, so I proceeded to explain to them how this works, as I was asked to do so in this language, so I should explain properly how this is handled in the language in question, and was met with skepticism, and doubt.

<?php
$element = array();
for ($i = 7; $i >= 0; --$i) {
  $element[$i] = $i;
}

unset($element[4]);

$element[4] = 4;

print implode(' ', $element);

In this case the result would be:

7 6 5 3 2 1 0 4

So what actually happened, because this is not an array and it’s an ordered map the result is not what you would expect.

If you actually want the keys to be sorted you will need to use ksort function.

The real index ordering is inaccessible to us, if you actually write this in JavaScript, Python, Ruby, Perl, Java, or C# you will get:

0 1 2 3 4 5 6 7

So the reason why is this happening is because this is an ordered map, but lets actually see what goes under the hood in PHP.

Basically everything in PHP is a HASH TABLE, not only the implementation of PHP arrays, but also they are used to store object properties, methods, functions, variables, … (everything)

And because the hash table is so fundamental to PHP, you should know what it actually is, before proceeding with PHP, at least that is what I think.

Any variable in PHP is defined by zval, and it’s value by zval_value, lets see the data structures.

These are taken from the PHP source code.

PHP_5_5/Zend/zend_hash.h#53

typedef struct _hashtable {
	uint nTableSize;
	uint nTableMask;
	uint nNumOfElements;
	ulong nNextFreeElement;
	Bucket *pInternalPointer;	/* Used for element traversal */
	Bucket *pListHead;
	Bucket *pListTail;
	Bucket **arBuckets;
	dtor_func_t pDestructor;
	zend_bool persistent;
	unsigned char nApplyCount;
	zend_bool bApplyProtection;
#if ZEND_DEBUG
	int inconsistent;
#endif
} HashTable;

PHP_5_5/Zend/zend.h#_zval_struct

struct _zval_struct {
    /* Variable information */
    zvalue_value value;     /* value */
    zend_uint refcount__gc;
    zend_uchar type;    /* active type */
    zend_uchar is_ref__gc;
};

PHP_5_5/Zend/zend.h#_zvalue_value

typedef union _zvalue_value {
    long lval;                  /* long value */
    double dval;                /* double value */
    struct {
        char *val;
        int len;
    } str;
    HashTable *ht;              /* hash table value */
    zend_object_value obj;
} zvalue_value;

Just a simple explanation, for those of you that don’t know C, you see the union there, that basically means that this is not actually a structure, but a single type.

In C, variables are just labels for raw memory address.

As you can see in the data structures above, we keep track of what primitive type is the value.

I won’t delve deep into explaining this, it will take quite a while to do so, and maybe I will do so in a future article.

If we actually have this code

<?php
$a = 1;
$b =& $a;
unset($a);

In this case $a will be unset, but $b will still remain the same with the value 1.

Now let’s see this too

<?php
$a = 1;
$b = &$a;
$a = 2;
echo "$a = $b";

In the previous case when you unset, the value actually remains in $b, but in this case, if we change $a, $b will also change.

Well we strayed quite a bit from arrays, so let’s finish it here for PHP.

Java

In the heap (not on the stack), the array is a block of memory made of all the elements together, each element in the arrays is identified by a zero based index [0,1,2,3,….].

There are ways of putting objects and arrays onto the stack instead, it’s called Escape analysis, you can read some info on this link

The Java language does not offer any way to explicitly allocate an object on the stack, but this fact doesn’t prevent JVMs from still using stack allocation where appropriate. JVMs can use a technique called escape analysis, by which they can tell that certain objects remain confined to a single thread for their entire lifetime, and that lifetime is bounded by the lifetime of a given stack frame. Such objects can be safely allocated on the stack instead of the heap. Even better, for small objects, the JVM can optimize away the allocation entirely and simply hoist the object’s fields into registers.

The length of an array in Java cannot be changed if created with new, let’s take a look at this example.

type arr[]; // declare the array
arr = new type[10]; // will initialize the array to 10 elements, and store the pointer

type can be a primitive or an object.

Lets see the previous example in PHP, with primitives without using Collections, or Lists.

public class Test {

    public static void main(String args[]) {
        
		int[] arr = new int[8];
		
		for(int index = arr.length - 1; index >= 0; index--) {
			arr[index] = index;
		}
		
		for(int index = 0; index < arr.length; index++) {
			System.out.print(arr[index]);
		}    
    }

}

The output will be

0 1 2 3 4 5 6 7

Even though we inserted them in reverse the order is still correct.

If you want to remove an array, you need to follow create a new array and copy it into the old one, while removing the element you don’t want.

You can use System.copyarray to do so

C

Arrays are allocated as a fixed number of contiguous elements, and there is no way to actually remove the memory used by an individual element in the array, but the elements can be shifted to fill in the hole made by the removal of the element.

As you know from above that statical arrays can not be resized, but dynamically allocated ones can be resized with realloc(), and what this will do is move the array to a new memory location, that can accomodate the whole array.

Javascript

In javascript the arrays can be sparse arrays and dense array. Javascript arrays are dynamic. So any implementation has to account for variable length of the array.

The Javascript Array wraps a simple C array allocated with a fixed number of elements. (a flat array) The elements in the C array are resized everytime access to a location beyond the current maximum length is required. Hence insertion could be expensive.

When the runtime detects that the array is Sparse, it is implemented in a similar way to an object. So instead of maintaining a contiguous array, a key/value map is built. In case of an array since the indexes are integers, this map can be efficiently implemented using a balanced Binary Search Tree or a similar structure. The performance of a sparse array is likely similar to the performance of an object.

The arrays are initialized with Flat arrays and are then later replaced with Sparse array map if necessary.

Even with the above general ideas, there is still so much leeway for browsers to tweak their implementations – the heuristic for deciding if the array is sparse/dense, internal data structures used etc.

The key lessons though, from understanding these differences are:

Never use an object, when you need an array
Never initialize arrays backwards (this will force create sparse array which is less efficient than dense arrays)

Lets look at some examples

Sparse arrays

When you define the length of the array, you are not actually allocating the entries up front, but you are just allocates a slot for all the entries.

So if we do this

var arr = new Array(10);
arr.forEach(function(v, k) {
 console.log(v, k);
});

The array is actually

[undefined × 10]

The keys are allocated but with no value, and the for cycle will actually not output anything.

Dense arrays

var arr = Array.apply(null, Array(10));
arr.forEach(function(v, k) {
 console.log(v, k);
});

The array now is

[undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined]

And as you can see now the array is now defined.

One thing that we can use with this is

Array.apply(null, Array(10)).map(function (x,i) { return i })

And this will create for you an array of items

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Lets see the example from up, and compare it with PHP.

arr = [];
for(i=7; i >= 0; i--) {
   arr[i]=i;
} 
console.log(arr);

And the output will be

0 1 2 3 4 5 6 7

Ending

Well I think it’s enough to describe some of the intricacies of arrays between programing languages, we can probably go into more detail, but it should be sufficient.

Well this turned out to be quite long.

If you made it to the end, leave a comment, or your thoughts on this.

Getting started with gradle and java

Thu, 02 Oct 2014 23:26:00 +0000

I always used ant with Apache Ivy or maven to build java applications, so I thought there has to be something better and easier to build the application instead of using them. So I decided to give gradle a go.

To build a project with gradle is quite simple, and unlike ant and maven, managing dependencies is really easy.

build.gradle

apply plugin: 'java'
apply plugin: 'eclipse'

version = '1.0'

eclipse {
    classpath {
       downloadSources=true
       downloadJavadoc=true
    }
}

repositories {
    mavenCentral()
}

dependencies {
    compile 'log4j:log4j:1.2.17'
    compile 'com.google.guava:guava:18.0'
    compile 'io.netty:netty-all:4.0.23.Final'
    testCompile 'junit:junit:4.11'
}

jar {
    from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
    manifest { 
        attributes 'Implementation-Title': '', 'Implementation-Version': version
        attributes 'Main-Class': 'com.matoski.blog.demo.Hello'
    }
}

src/main/java/com/matoski/blog/demo/Hello.java

package com.matoski.blog.demo;

public class Hello {

	public static void main(String[] args) {
		System.out.println("Hello");
	}
}

If you run the build command gradle build it will build the project and create a jar file so we can run it.

# gradle build
:compileJava
:processResources UP-TO-DATE
:classes
:jar
:assemble
:compileTestJava UP-TO-DATE
:processTestResources
:testClasses
:test UP-TO-DATE
:check UP-TO-DATE
:build

BUILD SUCCESSFUL

Total time: 12.48 secs

And now we can run the application easily by running

# java -jar build/libs/hello.jar
Hello

It was really simple to build everything with gradle.

Express4 + Mongoose + JSON Web Token Authentication

Sat, 20 Sep 2014 09:47:00 +0000

Authentication is part of almost every system, even if it is in node.js, Express, Angular.JS, PHP, Perl, Ruby, or any other languages you are using. Dealing with authentication is a must for most of the systems.

This article is quite long, so be prepared.

Tutorial Resources

How we used to do authentication

As we know HTTP protocol is stateless, so it cannot remember anything between requests, as it forgets everything on the next request.

Imagine if you need to login in on every request you make to the page, which is a real pain.

How did we solve this

Sessions: We have to store our sessions on the server, and if we have multiple server then we need to synchronize the sessions between the servers, we can use redis to make it easier to share the sessions. But with no sessions we don’t have a problem.
Mobile: From my experience native mobile apps have problems working with cookies. If we need to query an API maybe session is not the best way to do it.
CSRF: If we use cookies for authentications, we will have to implement CSRF, for more details visit this link
CORS: Have you tried using cookies with CORS? For more details visit this link

JSON Web Token

Good thing about JWT is that it doesn’t use sessions, meaning has no problems with CSRF, works excellent with CORS, Mobile

So the flow is: 1. Client Login 2. Client receives a token 3. Client does request with the token 4. Token gets decoded on the server, and you get the information stored in the token - In here you can verify if the user has access for this resource, this will simplify ACL - If the token is invalid return 401 5. With the data the server has it will decide if it will let the user get data or return a 401 - I can query database and return the data that the user requested if he has privileges - I can update if the user has privileges

So one important thing is how and where do we store the token? * localStorage * sessionStorage * in the application that does the requests

Another thing you can also do is share your token with a different client and he can login with that token too.

To create a token we will need a secret key, as long as we don’t share that key, no one can

If you are interested in more details you can read the IETF JSON Web Token draft.

Implementation

Lets see how we would go about implementing this.

We can use express-generator, to generate a skeleton for the application, but in our case we wouldn’t need one as we are only gonna focus on creating an authentication mechanism, without a front-end

Preparation

So let’s run the following commands to create the necessary structure for the project

mkdir express-jwt-auth
mkdir express-jwt-auth/keys
mkdir express-jwt-auth/routes
mkdir express-jwt-auth/models
mkdir express-jwt-auth/errors

We need to prepare the package.json so we can install all the dependencies

{
    "name": "express-jwt-auth",
    "version": "0.0.1",
    "private": true,
    "main": "app.js",
    "dependencies": {
        "bcryptjs": "~2.0.2",
        "body-parser": "~1.0.0",
        "compression": "^1.0.11",
        "cors": "^2.4.1",
        "debug": "~0.7.4",
        "express": "^4.2.0",
        "express-jwt": "^0.3.1",
        "express-unless": "*",
        "hiredis": "^0.1.17",
        "jsonwebtoken": "^0.4.1",
        "lodash": "~2.4.1",
        "mongoose": "~3.8.14",
        "morgan": "~1.2.2",
        "on-finished": "^2.1.0",
        "redis": "^0.12.1",
        "response-time": "~2.0.1"
    }
}

You can look up these packages to see what all of them do.

You need to install all the package, so issue the following command

npm install

Errors

First lets create the error files in errors directory, if you are asking why we would use this, it’s simple, we can pass the errors onto the next handler, and we can process all the errors and handle them in one places instead of handling them in each route separately.

errors/NotFoundError.js

"use strict";
function NotFoundError(code, error) {
    Error.call(this, typeof error === "undefined" ? undefined : error.message);
    Error.captureStackTrace(this, this.constructor);
    this.name = "NotFoundError";
    this.message = typeof error === "undefined" ? undefined : error.message;
    this.code = typeof code === "undefined" ? "404" : code;
    this.status = 404;
    this.inner = error;
}

NotFoundError.prototype = Object.create(Error.prototype);
NotFoundError.prototype.constructor = NotFoundError;

module.exports = NotFoundError;

errors/UnauthorizedAccessError.js

"use strict";
function UnauthorizedAccessError(code, error) {
    Error.call(this, error.message);
    Error.captureStackTrace(this, this.constructor);
    this.name = "UnauthorizedAccessError";
    this.message = error.message;
    this.code = code;
    this.status = 401;
    this.inner = error;
}

UnauthorizedAccessError.prototype = Object.create(Error.prototype);
UnauthorizedAccessError.prototype.constructor = UnauthorizedAccessError;

module.exports = UnauthorizedAccessError;

We have created the errors, that we are gonna use in the application.

Main App

Now we need to create the main file app.js in the root of the directory with the following contents

app.js

"use strict";

var debug = require('debug')('app:' + process.pid),
    path = require("path"),
    fs = require("fs"),
    http_port = process.env.HTTP_PORT || 3000,
    https_port = process.env.HTTPS_PORT || 3443,
    jwt = require("express-jwt"),
    config = require("./config.json"),
    mongoose_uri = process.env.MONGOOSE_URI || "localhost/express-jwt-auth",
    onFinished = require('on-finished'),
    NotFoundError = require(path.join(__dirname, "errors", "NotFoundError.js")),
    utils = require(path.join(__dirname, "utils.js")),
    unless = require('express-unless');

debug("Starting application");

debug("Loading Mongoose functionality");
var mongoose = require('mongoose');
mongoose.set('debug', true);
mongoose.connect(mongoose_uri);
mongoose.connection.on('error', function () {
    debug('Mongoose connection error');
});
mongoose.connection.once('open', function callback() {
    debug("Mongoose connected to the database");
});

debug("Initializing express");
var express = require('express'), app = express();

debug("Attaching plugins");
app.use(require('morgan')("dev"));
var bodyParser = require("body-parser");
app.use(bodyParser.json());
app.use(bodyParser.urlencoded());
app.use(require('compression')());
app.use(require('response-time')());

app.use(function (req, res, next) {

    onFinished(res, function (err) {
        debug("[%s] finished request", req.connection.remoteAddress);
    });

    next();

});

var jwtCheck = jwt({
    secret: config.secret
});
jwtCheck.unless = unless;

app.use(jwtCheck.unless({path: '/api/login' }));
app.use(utils.middleware().unless({path: '/api/login' }));

app.use("/api", require(path.join(__dirname, "routes", "default.js"))());

// all other requests redirect to 404
app.all("*", function (req, res, next) {
    next(new NotFoundError("404"));
});

// error handler for all the applications
app.use(function (err, req, res, next) {

    var errorType = typeof err,
        code = 500,
        msg = { message: "Internal Server Error" };

    switch (err.name) {
        case "UnauthorizedError":
            code = err.status;
            msg = undefined;
            break;
        case "BadRequestError":
        case "UnauthorizedAccessError":
        case "NotFoundError":
            code = err.status;
            msg = err.inner;
            break;
        default:
            break;
    }

    return res.status(code).json(msg);

});

debug("Creating HTTP server on port: %s", http_port);
require('http').createServer(app).listen(http_port, function () {
    debug("HTTP Server listening on port: %s, in %s mode", http_port, app.get('env'));
});

debug("Creating HTTPS server on port: %s", https_port);
require('https').createServer({
    key: fs.readFileSync(path.join(__dirname, "keys", "server.key")),
    cert: fs.readFileSync(path.join(__dirname, "keys", "server.crt")),
    ca: fs.readFileSync(path.join(__dirname, "keys", "ca.crt")),
    requestCert: true,
    rejectUnauthorized: false
}, app).listen(https_port, function () {
    debug("HTTPS Server listening on port: %s, in %s mode", https_port, app.get('env'));
});

So let’s see what happens here: * We connect to Mongo database via Mongoose * We add various plugins to extend Express * We initialize JSON Web Token library and give it a secret key * We exclude JSON Web Token verification for /api/login * We hook the default route, for login, logout, verify routes * We create handler for 404 messages * We create handler for Error messages * We start HTTP and HTTPS version on the defined ports

Models

Now let’s create the models/user.js file, this will serve us as a way to store the users and the password, and then authenticate against it later on

models/user.js

"use strict";

var mongoose = require('mongoose'),
    bcrypt = require("bcryptjs"),
    Schema = mongoose.Schema;

var UserSchema = new Schema({

    username: {
        type: String,
        unique: true,
        required: true
    },

    password: {
        type: String,
        required: true
    }

}, {
    toObject: {
        virtuals: true
    }, toJSON: {
        virtuals: true
    }
});

UserSchema.pre('save', function (next) {
    var user = this;
    if (this.isModified('password') || this.isNew) {
        bcrypt.genSalt(10, function (err, salt) {
            if (err) {
                return next(err);
            }
            bcrypt.hash(user.password, salt, function (err, hash) {
                if (err) {
                    return next(err);
                }
                user.password = hash;
                next();
            });
        });
    } else {
        return next();
    }
});

UserSchema.methods.comparePassword = function (passw, cb) {
    bcrypt.compare(passw, this.password, function (err, isMatch) {
        if (err) {
            return cb(err);
        }
        cb(null, isMatch);
    });
};

module.exports = mongoose.model('User', UserSchema);

As you can see, before saving the model into database, the password is hashed and we cannot retrieve it.

Routes

Now lets define the routes, we will use this for logging in or logging out of the system, also to verify if we are logged in or not

First lets create the necessary utilities to be able to log in and log out

utils.js

"use strict";

var debug = require('debug')('app:utils:' + process.pid),
    path = require('path'),
    util = require('util'),
    redis = require("redis"),
    client = redis.createClient(),
    _ = require("lodash"),
    config = require("./config.json"),
    jsonwebtoken = require("jsonwebtoken"),
    TOKEN_EXPIRATION = 60,
    TOKEN_EXPIRATION_SEC = TOKEN_EXPIRATION * 60,
    UnauthorizedAccessError = require(path.join(__dirname, 'errors', 'UnauthorizedAccessError.js'));

client.on('error', function (err) {
    debug(err);
});

client.on('connect', function () {
    debug("Redis successfully connected");
});

module.exports.fetch = function (headers) {
    if (headers && headers.authorization) {
        var authorization = headers.authorization;
        var part = authorization.split(' ');
        if (part.length === 2) {
            var token = part[1];
            return part[1];
        } else {
            return null;
        }
    } else {
        return null;
    }
};

module.exports.create = function (user, req, res, next) {

    debug("Create token");

    if (_.isEmpty(user)) {
        return next(new Error('User data cannot be empty.'));
    }

    var data = {
        _id: user._id,
        username: user.username,
        access: user.access,
        name: user.name,
        email: user.email,
        token: jsonwebtoken.sign({ _id: user._id }, config.secret, {
            expiresInMinutes: TOKEN_EXPIRATION
        })
    };

    var decoded = jsonwebtoken.decode(data.token);

    data.token_exp = decoded.exp;
    data.token_iat = decoded.iat;

    debug("Token generated for user: %s, token: %s", data.username, data.token);

    client.set(data.token, JSON.stringify(data), function (err, reply) {
        if (err) {
            return next(new Error(err));
        }

        if (reply) {
            client.expire(data.token, TOKEN_EXPIRATION_SEC, function (err, reply) {
                if (err) {
                    return next(new Error("Can not set the expire value for the token key"));
                }
                if (reply) {
                    req.user = data;
                    next(); // we have succeeded
                } else {
                    return next(new Error('Expiration not set on redis'));
                }
            });
        }
        else {
            return next(new Error('Token not set in redis'));
        }
    });

    return data;

};

module.exports.retrieve = function (id, done) {

    debug("Calling retrieve for token: %s", id);

    if (_.isNull(id)) {
        return done(new Error("token_invalid"), {
            "message": "Invalid token"
        });
    }

    client.get(id, function (err, reply) {
        if (err) {
            return done(err, {
                "message": err
            });
        }

        if (_.isNull(reply)) {
            return done(new Error("token_invalid"), {
                "message": "Token doesn't exists, are you sure it hasn't expired or been revoked?"
            });
        } else {
            var data = JSON.parse(reply);
            debug("User data fetched from redis store for user: %s", data.username);

            if (_.isEqual(data.token, id)) {
                return done(null, data);
            } else {
                return done(new Error("token_doesnt_exist"), {
                    "message": "Token doesn't exists, login into the system so it can generate new token."
                });
            }

        }

    });

};

module.exports.verify = function (req, res, next) {

    debug("Verifying token");

    var token = exports.fetch(req.headers);

    jsonwebtoken.verify(token, config.secret, function (err, decode) {

        if (err) {
            req.user = undefined;
            return next(new UnauthorizedAccessError("invalid_token"));
        }

        exports.retrieve(token, function (err, data) {

            if (err) {
                req.user = undefined;
                return next(new UnauthorizedAccessError("invalid_token", data));
            }

            req.user = data;
            next();

        });

    });
};

module.exports.expire = function (headers) {

    var token = exports.fetch(headers);

    debug("Expiring token: %s", token);

    if (token !== null) {
        client.expire(token, 0);
    }

    return token !== null;

};

module.exports.middleware = function () {

    var func = function (req, res, next) {

        var token = exports.fetch(req.headers);

        exports.retrieve(token, function (err, data) {

            if (err) {
                req.user = undefined;
                return next(new UnauthorizedAccessError("invalid_token", data));
            } else {
                req.user = _.merge(req.user, data);
                next();
            }

        });
    };

    func.unless = require("express-unless");

    return func;

};

module.exports.TOKEN_EXPIRATION = TOKEN_EXPIRATION;
module.exports.TOKEN_EXPIRATION_SEC = TOKEN_EXPIRATION_SEC;

debug("Loaded");

What does this file do, it is responsible for loading, saving the token into redis, and when verifying the token to make sure it is present in redis too. By storing the token in redis, we get and extra option, like invalidating tokens, if we just remove it from redis, even if the token is valid but not present, the API will return a 401 error.

routes/default.js

"use strict";

var debug = require('debug')('app:routes:default' + process.pid),
    _ = require("lodash"),
    util = require('util'),
    path = require('path'),
    bcrypt = require('bcryptjs'),
    utils = require("../utils.js"),
    Router = require("express").Router,
    UnauthorizedAccessError = require(path.join(__dirname, "..", "errors", "UnauthorizedAccessError.js")),
    User = require(path.join(__dirname, "..", "models", "user.js")),
    jwt = require("express-jwt");

var authenticate = function (req, res, next) {

    debug("Processing authenticate middleware");

    var username = req.body.username,
        password = req.body.password;

    if (_.isEmpty(username) || _.isEmpty(password)) {
        return next(new UnauthorizedAccessError("401", {
            message: 'Invalid username or password'
        }));
    }

    process.nextTick(function () {

        User.findOne({
            username: username
        }, function (err, user) {

            if (err || !user) {
                return next(new UnauthorizedAccessError("401", {
                    message: 'Invalid username or password'
                }));
            }

            user.comparePassword(password, function (err, isMatch) {
                if (isMatch && !err) {
                    debug("User authenticated, generating token");
                    utils.create(user, req, res, next);
                } else {
                    return next(new UnauthorizedAccessError("401", {
                        message: 'Invalid username or password'
                    }));
                }
            });
        });

    });


};

module.exports = function () {

    var router = new Router();

    router.route("/verify").get(function (req, res, next) {
        return res.status(200).json(undefined);
    });

    router.route("/logout").get(function (req, res, next) {
        if (utils.expire(req.headers)) {
            delete req.user;
            return res.status(200).json({
                "message": "User has been successfully logged out"
            });
        } else {
            return next(new UnauthorizedAccessError("401"));
        }
    });

    router.route("/login").post(authenticate, function (req, res, next) {
        return res.status(200).json(req.user);
    });

    router.unless = require("express-unless");

    return router;
};

debug("Loaded");

Now let’s try to login to the system, we can use curl to do this:

# curl -v -d "username=demo&password=demo"  http://127.0.0.1:3000/api/login

* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> POST /api/login HTTP/1.1
> User-Agent: curl/7.37.1
> Host: 127.0.0.1:3000
> Accept: */*
> Content-Length: 27
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 27 out of 27 bytes
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: application/json; charset=utf-8
< Content-Length: 281
< X-Response-Time: 362.181ms
< Vary: Accept-Encoding
< Date: Fri, 05 Sep 2014 06:07:50 GMT
< Connection: keep-alive
< 
* Connection #0 to host 127.0.0.1 left intact
{
  "_id": "540951d773d8bf915726de69",
  "username": "demo",
  "token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJfaWQiOiI1NDA5NTFkNzczZDhiZjkxNTcyNmRlNjkiLCJpYXQiOjE0MDk4OTcyNzAsImV4cCI6MTQwOTkwMDg3MH0.NmzXaVUWpAaZLnq4lpsy_HlV6GqW2leOkOqyrvYku-U",
  "token_exp": 1409900870,
  "token_iat": 1409897270
}

As you can see we got a token back now, and we can use this to query the API. What we are doing is we are storing the user id in the token so when we decode it we can use the id to query the database for details, as we are storing the token and the user data in redis, we can use this id to get all the information about the user from Redis instead of querying the data all the time.

token is the actual token
token_exp until when is the token valid (unix timestamp)
token_iat is when the token was created (unix timestamp)

/api/verify

Now let’s see about verifying if we are logged in or not

# curl -v  http://127.0.0.1:3000/api/verify
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> GET /api/verify HTTP/1.1
> User-Agent: curl/7.37.1
> Host: 127.0.0.1:3000
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< X-Powered-By: Express
< Content-Type: application/json
< X-Response-Time: 2.777ms
< Vary: Accept-Encoding
< Date: Fri, 05 Sep 2014 06:10:37 GMT
< Connection: keep-alive
< Transfer-Encoding: chunked
< 
* Connection #0 to host 127.0.0.1 left intact

You can see we got 401 response, we didn’t supply the token, let’s add the token

# curl -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJfaWQiOiI1NDA5NTFkNzczZDhiZjkxNTcyNmRlNjkiLCJpYXQiOjE0MDk4OTcyNzAsImV4cCI6MTQwOTkwMDg3MH0.NmzXaVUWpAaZLnq4lpsy_HlV6GqW2leOkOqyrvYku-U" -v  http://127.0.0.1:3000/api/verify
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> GET /api/verify HTTP/1.1
> User-Agent: curl/7.37.1
> Host: 127.0.0.1:3000
> Accept: */*
> Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJfaWQiOiI1NDA5NTFkNzczZDhiZjkxNTcyNmRlNjkiLCJpYXQiOjE0MDk4OTcyNzAsImV4cCI6MTQwOTkwMDg3MH0.NmzXaVUWpAaZLnq4lpsy_HlV6GqW2leOkOqyrvYku-U
> 
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: application/json
< X-Response-Time: 3.722ms
< Vary: Accept-Encoding
< Date: Fri, 05 Sep 2014 06:12:03 GMT
< Connection: keep-alive
< Transfer-Encoding: chunked
< 
* Connection #0 to host 127.0.0.1 left intact

Now we got 200, which means the token is valid

/api/logout

Same applies for logout, you cannot logout unless you are logged in, if you try to do it without the token you will get 401.

curl -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJfaWQiOiI1NDA5NTFkNzczZDhiZjkxNTcyNmRlNjkiLCJpYXQiOjE0MDk4OTcyNzAsImV4cCI6MTQwOTkwMDg3MH0.NmzXaVUWpAaZLnq4lpsy_HlV6GqW2leOkOqyrvYku-U" -v  http://127.0.0.1:3000/api/logout
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> GET /api/logout HTTP/1.1
> User-Agent: curl/7.37.1
> Host: 127.0.0.1:3000
> Accept: */*
> Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJfaWQiOiI1NDA5NTFkNzczZDhiZjkxNTcyNmRlNjkiLCJpYXQiOjE0MDk4OTcyNzAsImV4cCI6MTQwOTkwMDg3MH0.NmzXaVUWpAaZLnq4lpsy_HlV6GqW2leOkOqyrvYku-U
> 
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: application/json; charset=utf-8
< Content-Length: 51
< ETag: W/"33-3789284850"
< X-Response-Time: 4.644ms
< Vary: Accept-Encoding
< Date: Fri, 05 Sep 2014 06:14:20 GMT
< Connection: keep-alive
< 
* Connection #0 to host 127.0.0.1 left intact
{"message":"User has been successfully logged out"}

Creating a user (create_user.js)

var path = require("path"),
    config = require("./config.json"),
    User = require(path.join(__dirname, "models", "user.js")),
    mongoose_uri = process.env.MONGOOSE_URI || "localhost/express-jwt-auth";

var args = process.argv.slice(2);

var username = args[0];
var password = args[1];

if (args.length < 2) {
    console.log("usage: node %s %s %s", path.basename(process.argv[1]), "user", "password");
    process.exit();
}

console.log("Username: %s", username);
console.log("Password: %s", password);

console.log("Creating a new user in Mongo");


var mongoose = require('mongoose');
mongoose.set('debug', true);
mongoose.connect(mongoose_uri);
mongoose.connection.on('error', function () {
    console.log('Mongoose connection error', arguments);
});
mongoose.connection.once('open', function callback() {
    console.log("Mongoose connected to the database");

    var user = new User();

    user.username = username;
    user.password = password;

    user.save(function (err) {
        if (err) {
            console.log(err);
        } else {
            console.log(user);
        }
        process.exit();
    });

});

So to create a user just execute the following command

# node create_user.js demo demo

Username: demo
Password: demo
Creating a new user in Mongo
Mongoose: users.ensureIndex({ username: 1 }) { safe: undefined, background: true, unique: true }  
Mongoose connected to the database
Mongoose: users.insert({ __v: 0, _id: ObjectId("540951d773d8bf915726de69"), username: 'demo', password: '$2a$10$7t4Uz6WUapkmvr.uN1PVkuHfc6JcuMuWiElfv6CFMi/GESe1qSAt2' }) {}  
{ __v: 0,
  password: '$2a$10$7t4Uz6WUapkmvr.uN1PVkuHfc6JcuMuWiElfv6CFMi/GESe1qSAt2',
  username: 'demo',
  _id: 540951d773d8bf915726de69,
  id: '540951d773d8bf915726de69' }

This will create a user demo that you can use to log in

Conclusion

This approach will enable you a lot of flexibility with login/logout, and API architecture.

Any modification to the token will invalidate it, also the only way to do modifications to the token is with a secret key, but this is not something we share with other people.

Also this approach sets a expiry date, in our case 60 min, also that information is saved into the payload, so on the frontend, we can know when the token expires, and reissue a new token before expiry, or user has to re-login into the system to continue using it.

Using JSON Web Token should be done over SSL preferably as you are sending the token on every request.

If you want even more security you can change the secret key on every user login, or use a different secret key for every user.

How to generate self-signed certificate for usage in Express4 or Node.js HTTP

Tue, 09 Sep 2014 11:05:00 +0000

I needed to generate a self-signed certificate for usage with node.js and express, since I don’t want to buy a certificate for just trying out and playing with it.

Let’s figure out how to do it.

You can also take a look at the following YouTube Video

Make sure you have install openssl, if you haven’t install it

RHEL/CentOS systems

yum install openssl

Debian

apt-get install openssl

To be able to use SSL you need to generate

Certificate Authority
Server Certificate

Before we generate anything we need to generate a secure pass phrase

# pwgen 50 1 -s > passphrase
# cat passphrase
TNOojiJgDeqP1WwUYflXzBpbfZyl1vkAiuoXoikXRPQ9d1VBkC

So that is our secure pass phrase, and whenever it asks you for a pass phrase you can just copy paste the one we generated above.

Let’s generate Certificate Authority first

Private Key

# openssl genrsa -des3 -out ca.key 1024
Generating RSA private key, 1024 bit long modulus
...++++++
...++++++
e is 65537 (0x10001)
Enter pass phrase for ca.key:
Verifying - Enter pass phrase for ca.key:

Certificate Signing Request

openssl req -new -key ca.key -out ca.csr
Enter pass phrase for ca.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Signing the certificate

# openssl x509 -req -days 365 -in ca.csr -out ca.crt -signkey ca.key
Signature ok
subject=/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
Getting Private key
Enter pass phrase for ca.key:

Now let’s generate the Server Certificate

Private Key with pass phrase

# openssl genrsa -des3 -out server.key 1024
Generating RSA private key, 1024 bit long modulus
................................++++++
.....++++++
e is 65537 (0x10001)
Enter pass phrase for server.key:
Verifying - Enter pass phrase for server.key:

Certificate Signing Request

# openssl req -new -key server.key -out server.csr
Enter pass phrase for server.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Private Key without pass phrase This will remove the pass phrase from the key, this step is crucial without this it will not work

# cp server.key server.key.passphrase
# openssl rsa -in server.key.passphrase -out server.key
openssl rsa -in server.key.passphrase -out server.key
Enter pass phrase for server.key.passphrase:
writing RSA key

Signing the certificate

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
Signature ok
subject=/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
Getting Private key

These are the files we have now

ls -la
total 36
drwxr-xr-x  2 user user 4096 Sep  5 16:19 .
drwxr-xr-x 12 user user 4096 Sep  5 16:09 ..
-rw-r--r--  1 user user  757 Sep  5 16:12 ca.crt
-rw-r--r--  1 user user  603 Sep  5 16:10 ca.csr
-rw-r--r--  1 user user  963 Sep  5 16:09 ca.key
-rw-r--r--  1 user user  757 Sep  5 16:19 server.crt
-rw-r--r--  1 user user  603 Sep  5 16:16 server.csr
-rw-r--r--  1 user user  887 Sep  5 16:18 server.key
-rw-r--r--  1 user user  951 Sep  5 16:17 server.key.passphrase

There you go now we have everything needed, lets see how we can create a HTTPS server with node.js

var https = require('https'),
    fs = require('fs'), 
    express = require('express'), 
    app = express();
    
var secureServer = https.createServer({
    key: fs.readFileSync('./ssl/server.key'),
    cert: fs.readFileSync('./ssl/server.crt'),
    ca: fs.readFileSync('./ssl/ca.crt'),
    requestCert: true,
    rejectUnauthorized: false
}, app).listen('8443', function() {
    console.log("Secure Express server listening on port 8443");
});

Emergency reboot/shutdown using SysRq

Mon, 08 Sep 2014 08:07:00 +0000

As you know linux implements some type of mechanism to gracefully shutdown and reboot, this means the daemons are stopping, usually linux stops them one by one, the file cache is synced to disk.

But what sometimes happens is that the system will not reboot or shutdown no mater how many times you issue the shutdown or reboot command.

If the server is close to you, you can always just do a physical reset, but what if it’s far away from you, where you can’t reach it, sometimes it’s not feasible, why if the OpenSSH server crashes and you cannot log in again in the system.

If you ever find yourself in a situation like that, there is another option to force the system to reboot or shutdown.

The magic SysRq key is a key combination understood by the Linux kernel, which allows the user to perform various low-level commands regardless of the system’s state. It is often used to recover from freezes, or to reboot a computer without corrupting the filesystem.

Description	QWERTY
Immediately reboot the system, without unmounting or syncing filesystems	b
Sync all mounted filesystems	s
Shut off the system	o
Send the SIGKILL signal to all processes except init	i

So if you are in a situation where you cannot reboot or shutdown the server, you can force an immediate reboot by issuing

echo 1 > /proc/sys/kernel/sysrq 
echo b > /proc/sysrq-trigger

If you want you can also force a sync before rebooting by issuing these commands

echo 1 > /proc/sys/kernel/sysrq 
echo s > /proc/sysrq-trigger
echo b > /proc/sysrq-trigger

These are called magic commands, and they’re pretty much synonymous with holding down Alt-SysRq and another key on older keyboards. Dropping 1 into /proc/sys/kernel/sysrq tells the kernel that you want to enable SysRq access (it’s usually disabled). The second command is equivalent to pressing *Alt-SysRq-b on a QWERTY keyboard.

If you want to keep SysRq enabled all the time, you can do that with an entry in your server’s sysctl.conf:

echo "kernel.sysrq = 1" >> /etc/sysctl.conf

rsyslog centralized logging

Thu, 04 Sep 2014 18:56:00 +0000

First of all logging is really important, and are a critical part of any system, they will give you insight into what a system is doing as well what happened.

Virtually every process running on a system generates logs in some form or another. When your system grows to multiple hosts, managing the logs and accessing them can get complicated.

Searching for a particular error across hundreds of log files on hundreds of servers is difficult without good tools.

A common approach to this problem is to setup a centralized logging solution so that multiple logs can be aggregated in a central location.

Most people use rsyslog or syslog-ng which are two syslog implementations. These daemons allow processes to send log messages to them and the syslog configuration determines how the are stored. In a centralized logging setup, a central syslog daemon is setup on your network and the client logging dameons are setup to forward messages to the central daemon

Step 1 - Installation

If you don’t have rsyslog on the server for any reason you can install them easily.

RHEL/CentOS systems

yum install rsyslog

Debian

apt-get install rsyslog

Step 2 - Configuration

So what we need is for us to configure a server that will receive all the logs from remote servers, and configure the clients to send the logs to the server.

Lets configure the server first.

The configuration file is found in /etc/rsyslog.conf, open it with your favorite editor, and lets edit the file, and we need to decide if we gonna use TCP or UDP, if you are not sure, just stick with TCP.

If you want to use TCP, add the following lines in /etc/rsyslog.conf

$ModLoad imtcp
$InputTCPServerRun 514

As for UDP

$ModLoad imudp
$UDPServerRun 514

Yes one can have both UDP and TCP requests on same port as each connection is identified by ( source IP ,Destination IP, Source Port, Destination Port, PROTOCOL) as protocol can be TCP or UDP both connections can be differentiated.

RHEL/CentOS systems, Debian

service rsyslog restart

Now let’s check if the server is listening on the ports we have specified.

TCP

$ lsof -i :514
rsyslogd 7004 root    5u  IPv4 118878      0t0  TCP *:shell (LISTEN)
rsyslogd 7004 root    6u  IPv6 118879      0t0  TCP *:shell (LISTEN)

UDP

$ lsof -i :514
rsyslogd 7004 root    5u  IPv4 118878      0t0  UDP *:514
rsyslogd 7004 root    6u  IPv6 118879      0t0  UDP *:514

So as you can see now we have rsyslog listening on both TCP and UDP on the port we have specified.

Lets configure the client now.

All you need to do, is just add the server to where you want to forward the logs.

*.*   @@Primary rsyslog server:514
*.*   @@Other Primary rsyslog server:514

What this does is, it sends the all the logs to both servers specified in the config file (/etc/rsyslog.conf)

You can use a selector for defining what logs you want to be sent.A selector processes all messages it receives (*.info;mail.none;authpriv.none;cron.none), and tries to forward every message to Primary rsyslog server

Another thing you can do is, add failover capability to your logging, but this will only work if the server is configured in TCP mode

/etc/rsyslog.conf

*.info;mail.none;authpriv.none;cron.none  @@<Primary rsyslog server>
$ActionExecOnlyWhenPreviousIsSuspended on
&@@<Address of secondary rsyslog server >
&@@<Address of third rsyslog server >
& /var/log/localbuffer
$ActionExecOnlyWhenPreviousIsSuspended off

So what happens is, if the primary rsyslog server cannot be reached, it will try on the secondary, and if the secondary can’t be reached it will try the third, and if the third is not reachable, it will write the data to /var/log/localbuffer, ideally we would never ever have to write to this file.

Step 3 - Firewall and SE Linux on the Server

If you are using SELinux you will have to allow traffic, just replace protocol with the correct one tcp, udp.

semanage -a -t syslogd_port_t -p <protocol> 514

If you are using iptables for firewall you will have to allow, correct one tcp, udp.

iptables -A INPUT -m state --state NEW -m <protocol> -p <protocol> --dport 514 -j ACCEPT

So there you have it, now you have your logs in a singular place.

Having everything in one place will allow you to run a really nice tool like ossec.

In a future article I will take a look at other mechanisms for transferring logs.

Upgrade Plesk 11.5 to Plesk 12 on Debian Wheezy 7.6, fixing OpenDKIM and DK

Sat, 02 Aug 2014 10:38:00 +0000

I wanted to upgrade to the newer edition of Plesk as it has some interesting features for controlling outgoing mail that make my life easier. So I wanted to upgrade and ran into some weird issues during the upgrade.

After downloading the newest Parallels autoinstall script from their webpage, and running it produced the following error.

Components validation detected at least one important issue:
Parallels Panel pre-upgrade check...
dpkg-query: no packages found matching psa-atmail
dpkg-query: no packages found matching psa-horde
dpkg-query: no packages found matching libapache2-modsecurity
WARNING: IP address registered in Plesk is invalid or broken: xxx.xxx.xxx.xxx

The OS version details are as follow

# lbs_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 7.6 (wheezy)
Release:	7.6
Codename:	wheezy

The panel version is 11.5.30 Update #47

It was a little weird why is stuff missing, so I decided to check if the packages are in the repo.

So running this command will give you the list of all the packages in the repository.

apt-get install grep-dctrl
grep-dctrl -sPackage . /var/lib/apt/lists/autoinstall.plesk.com_debian_*_Packages | sort | uniq

So it looks like the 3 packages psa-atmail, psa-horde, libapache2-modsecurity are not present in the repository.

Turns out that atmail is deprecated, and it doesn’t mater, but the other 2 packages I want them.

And the autoinstaller adds the following lines into /etc/apt/sources.list

## This is temporary appended by Autoinstaller for 
## specify source of product's packages for APT.
deb http://autoinstall.plesk.com/debian/PSA_12.0.18 wheezy all
deb http://autoinstall.plesk.com/debian/SITEBUILDER_12.0.5 all all
deb http://autoinstall.plesk.com/debian/BILLING_12.0.18 all all
deb http://autoinstall.plesk.com/debian/NGINX_1.6.0 wheezy all

You can safely ignore these, for some reason after the upgrade, they appear in the repository, but not during the upgrade, but you won’t have any issues with this, at least I didn’t.

Lets see about the next error:

WARNING: IP address registered in Plesk is invalid or broken: xxx.xxx.xxx.xxx

Turns out that according the Plesk, this is an IP registered in the panel but it’s not present in the system, to see which ones are present and which ones are not we can just run the following commands.

# brctl show
bridge name	bridge id		STP enabled	interfaces
br0		8000.d43d7edb0649	no		eth0
							vnet0

And the info about the IP addresses.

# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet xx.xx.xx.xx/27 brd xx.xx.xx.xx scope global br0
    inet xx.xx.xx.xx/32 scope global br0
    inet xx.xx.xx.xx/32 scope global br0
    inet6 xxxx::xxxx:xxxx:xxxx:xxxx/64 scope link 
       valid_lft forever preferred_lft forever
4: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN qlen 500
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet6 xxxx::xxxx:xxxx:xxxx:xxxx/64 scope link 
       valid_lft forever preferred_lft forever

And the biggest problem is that the IP is there but plesk for some reason doesn’t detect it, as you can see I have 3 IP’s assigned to the network card, and it’s not detected one of them, and also I’m running a KVM machine with a dedicated IP address. And I’m using a bridge for all of them to be present in the system.

If this can happen with one IP, I don’t know, but it shouldn’t happen, as to why it showed an error with the bridge I have no idea.

So you can just remove the IP temporarily from the system and the run the installer again, then after the upgrade you can just rerun the Reread IP from Tools & Settings > IP Addresses.

Also one more thing if you have followed my guide about Setting up SPF + DK + DKIM with Postfix in Plesk 11.5 on Debian Wheezy, you will notice that DKIM signing is not working for some reason. But turns out to be really easy fix.

Open /etc/postfix/main.cf with your favorite editor, and edit the following line.

# OpenDKIM
smtpd_milters = , inet:127.0.0.1:8891, inet:127.0.0.1:12768

# OpenDKIM
smtpd_milters = , inet:127.0.0.1:12768, inet:127.0.0.1:8891

And that’s it for DKIM, now let’s see about DK (Domain Keys) you will notice that it’s not signing them anymore, well that is also a simple matter to fix, by doing the following.

If you have one or two domains it’s simple then, just run this for the domain.

/usr/local/psa/bin/domain_pref -u www.example.com -sign_outgoing_mail false
/usr/local/psa/bin/domain_pref -u www.example.com -sign_outgoing_mail true

This will re-enable the signing for the domain in question, it will take a little while until it’s actually working, but for some reason in Plesk you need to disable it then enable it again for it to work.

But in my case I had a lot of domains and I have to go through a big list of domains, so I wrote this script that will do this for all the domains.

#!/bin/bash
pass=`cat /etc/psa/.psa.shadow`
domains=`mysql --skip-column-names -u admin -p$pass -e 'select name from domains where parentDomainId = 0;' psa | tr '\t' ','`
for domain in $domains; do
echo "Processing: $domain"
/usr/local/psa/bin/domain_pref -u "$domain" -sign_outgoing_mail false
/usr/local/psa/bin/domain_pref -u "$domain" -sign_outgoing_mail true
done

And that’s it, now we have everything in place, and you can start using your panel with the new version.

**UPDATE: ** The packages are now present in the repository, so looks like they added them afterwards, don’t know why they weren’t present when I was upgrading but now they are so it’s OK, either way you can continue ignoring that info about the packages, but the IP has to be resolved before continuing.

tmux logging complete output to a file

Tue, 22 Jul 2014 22:48:00 +0000

I’ve needed to log the complete output of a node.js process I’ve been running, even though I’ve been logging everything, there are some situations where the process completely crashes and nothing was logged.

To log the output of tmux you can use pipes, but in my case I wanted something simpler and more easier and flexible.

I’ve been using the command script to log the output to a file, so we can adjust our approach a little so we can log everything automatically.

In the ~/.bash_profile we need to append the following lines

if [[ $TERM = "screen" ]] && [[ $(ps -p $PPID -o comm=) = "tmux" ]]; then
mkdir $HOME/logs 2> /dev/null
logname="$(date '+%d%m%Y%H%M%S').tmux.log"
script -f $HOME/logs/${logname}
exit
fi

What this does is, when you start tmux manually it will output all the screen output to a log file, but this is always not sufficient.

In my case I was using tmux to start some commands from the shell with new-session, and this approach for some reason it doesn’t work.

So what I did is, I created a script that I will run to start everything I need

#!/bin/bash
tmux new-session -s session -d -n "session" "bash"
tmux send -t session.0 ". $HOME/.bash_profile" ENTER
tmux send -t session.0 "<COMMAND 1 I NEED TO RUN>" ENTER
tmux send -t session.0 "<COMMAND 2 I NEED TO RUN>" ENTER
tmux send -t session.0 "<COMMAND 3 I NEED TO RUN>" ENTER
tmux send -t session.0 "<COMMAND n I NEED TO RUN>" ENTER

So what this does, it manually loads the .bash_profile script and we are good to go, we can log automatically again. This can also be used to load all the environments in tmux

RVM + GitLab on Debian

Fri, 27 Jun 2014 12:53:00 +0000

I needed to install my own GitLab on my server for my personal projects, so I don’t have to use GitHub and pay for it. This will require some work, but in the end you will save some money.

Before proceeding lets install some prerequisites.

apt-get install nginx pwgen openssh-server openssh-client sudo git-core build-essential zlib1g-dev libyaml-dev libssl-dev libgdbm-dev libreadline-dev libncurses5-dev libffi-dev curl openssh-server redis-server checkinstall libxml2-dev libxslt-dev libcurl4-openssl-dev libicu-dev logrotate python-docutils libcurl4-openssl-dev libexpat1-dev gettext libz-dev libssl-dev

Lets install RVM first before proceeding with installation of GitLab. I always install RVM as a global installation as I can use it with other users in the systems if necessary.

curl -sSL https://get.rvm.io | sudo bash -s stable

At the time of writing this article the latest version of GitLab is version 7. And the required version of ruby is 2.1.2 according to the requirements, so let’s install ruby

rvm install ruby-2.1.2
gem install bundler

Before continuing we need to set up SSH to allow for user environments, because if we don’t we will get the following message whenever we try to do something with git.

/usr/bin/env: ruby: no such file or directory

This is because there is no user environment when doing operations with git.

Let’s edit /etc/ssh/sshd_config and add the following to the end of the file.

PermitUserEnvironment yes

Execute the following commands with root

adduser --disabled-login --gecos 'GitLab' git

I prefer MySQL over PostgreSQL for GitLab so lets install MySQL for use with GitLab

apt-get install mysql-server mysql-client libmysqlclient-dev
mysql_secure_installation

Lets generate a password to use with the system.

pwgen 10 1
iePhee2lae

And then execute the following SQL

CREATE USER 'git'@'localhost' IDENTIFIED BY 'iePhee2lae'; 
SET storage_engine=INNODB;
CREATE DATABASE IF NOT EXISTS `gitlabhq_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`;
GRANT SELECT, LOCK TABLES, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON `gitlabhq_production`.* TO 'git'@'localhost';
FLUSH PRIVILEGES;

Before we proceed lets add Git user temporarily to sudoers (/etc/sudoers)

git     ALL=(ALL) NOPASSWD: ALL

Now we need to checkout GitLab, for more up to date installation instructions

su - git
git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 7-0-stable gitlab
cd /home/git/gitlab
cp config/gitlab.yml.example config/gitlab.yml
cp config/unicorn.rb.example config/unicorn.rb
cp config/initializers/rack_attack.rb.example config/initializers/rack_attack.rb
cp config/database.yml.mysql config/database.yml
sudo chown -R git log/
sudo chown -R git tmp/
sudo chmod -R u+rwX log/
sudo chmod -R u+rwX tmp/
sudo mkdir /home/git/gitlab-satellites
sudo chmod u+rwx,g=rx,o-rwx /home/git/gitlab-satellites
sudo chmod -R u+rwX tmp/pids/
sudo chmod -R u+rwX tmp/sockets/
sudo chmod -R u+rwX  public/uploads
git config --global user.name "GitLab"
git config --global user.email "example@example.com"
git config --global core.autocrlf input
sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab
sudo cp lib/support/init.d/gitlab.default.example /etc/default/gitlab
sudo cp lib/support/logrotate/gitlab /etc/logrotate.d/gitlab
sudo cp lib/support/nginx/gitlab /etc/nginx/sites-available/gitlab
sudo ln -s /etc/nginx/sites-available/gitlab /etc/nginx/sites-enabled/gitlab
sudo update-rc.d gitlab defaults 21

Now lets edit the configuration files and modify them according to our needs.

Make sure to change “localhost” to the fully-qualified domain name of your host serving GitLab where necessary

editor config/gitlab.yml

Change YOUR_SERVER_FQDN to the fully-qualified domain name of your host serving GitLab.

sudo editor /etc/nginx/sites-available/gitlab

Update username/password in config/database.yml. You only need to adapt the production settings (first part).

editor config/database.yml
chmod o-rwx config/database.yml

Enable cluster mode if you expect to have a high load instance. Ex. change amount of workers to 3 for 2GB RAM server.

editor config/unicorn.rb

Now lets install gitlab shell

cd /home/git/gitlab
bundle install --deployment --without development test postgres aws
bundle exec rake gitlab:shell:install[v1.9.6] REDIS_URL=redis://localhost:6379 RAILS_ENV=production
bundle exec rake gitlab:setup RAILS_ENV=production
bundle exec rake assets:precompile RAILS_ENV=production

Now let’s check if everything was installed correctly

bundle exec rake gitlab:env:info RAILS_ENV=production
bundle exec rake gitlab:check RAILS_ENV=production

Now lets start everything up

sudo service gitlab restart
sudo service nginx restart

KVM port forwarding with NAT network and libvirt on Debian

Tue, 17 Dec 2013 11:09:00 +0000

I needed to forward some ports from multiple KVM machines, I tried with iptables, but the problem is libvirt adds some rules of it’s own, and the rules were never in the correct place so it didn’t work.

Fortunately KVM supports hooks, and we can use them to do what we need.

You should be able to easily adapt this to any linux distro.

/etc/libvirt/hooks/daemon

Executed when the libvirt daemon is started, stopped, or reloads its configuration

/etc/libvirt/hooks/qemu

Executed when a QEMU guest is started, stopped, or migrated

/etc/libvirt/hooks/lxc

Executed when an LXC guest is started or stopped

For a more detailed list we you can check libvirt hooks on the libvirt home page.

For us we will need to use /etc/libvirt/hooks/qemu script, because that is the script which triggers whenever we have VM going through the following states

prepare (libvirt 0.9.0)
start (libvirt 0.8.0)
started (libvirt 0.9.13)
stopped (libvirt 0.8.0)
release (libvirt 0.9.0)
migrate (libvirt 0.9.11)
reconnect (libvirt 0.9.13)
attach (libvirt 0.9.13)

In our case we are interested in three events, and we will split them in two groups

stopped or reconnect
start or reconnect

We want to automatically add/remove rules to iptables on starting/stopping the VM, let’s see a structure we are gonna use to hold the mappings.

{
  '<guest name (as defined in xml)>': {
     'ip': '<private ip>',
     'publicip': '<public ip>',
     'portmap': 'all' |
            {
               '<proto>': [(<host port>, <guest port>), ...], ...
            }
     }, ...
}

From the definition it is obvious what should go where

proto is the protocol
portmap can be either all or the map

Also we should log all the actions taken by this script to syslog if we want to do checking or debuging, to see if everything is properly working.

I would gladly accept any improvements to this scripts, so feel free to improve this script, and I will update it, on the page. This script has been tested on Python 2.7.3 and should work on Python 3, as for older versions I don’t know.

So let’s take a look at the python script located under /etc/libvirt/hooks/qemu, make sure you execute chmod +x /etc/libvirt/hooks/qemu

#!/usr/bin/python
# -*- coding: utf-8 -*-

import sys
import os
import syslog

domain = sys.argv[1]
action = sys.argv[2]

iptables = '/sbin/iptables'

mapping = {
  "vm1": {
    "ip": "192.168.22.1",
    "publicip": "x.x.x.x",
    "portmap": {
      "tcp": [
        (3901, 80),
        (3701, 22),
        (31621, 31621)
      ]
    }
  }
}

syslog.syslog('Processing QEMU Signal, domain: ' + domain + ', action:'
              + action)

def rules(act, map_dict):
    if map_dict['portmap'] == 'all':
        cmd = \
            '{} -t nat {} PREROUTING -d {} -j DNAT --to {}'.format(iptables,
                act, map_dict['publicip'], map_dict['ip'])
        os.system(cmd)
        syslog.syslog(cmd)
        cmd = \
            '{} -t nat {} POSTROUTING -s {} -j SNAT --to {}'.format(iptables,
                act, map_dict['ip'], map_dict['publicip'])
        os.system(cmd)
        syslog.syslog(cmd)
        cmd = \
            '{} -t filter {} FORWARD -d {} -j ACCEPT'.format(iptables,
                act, map_dict['ip'])
        os.system(cmd)
        syslog.syslog(cmd)
        cmd = \
            '{} -t filter {} FORWARD -s {} -j ACCEPT'.format(iptables,
                act, map_dict['ip'])
        os.system(cmd)
        syslog.syslog(cmd)
    else:
        for proto in map_dict['portmap']:
            for portmap in map_dict['portmap'].get(proto):
                cmd = \
                    '{} -t nat {} PREROUTING -d {} -p {} --dport {} -j DNAT --to {}:{}'.format(
                    iptables,
                    act,
                    map_dict['publicip'],
                    proto,
                    str(portmap[0]),
                    map_dict['ip'],
                    str(portmap[1]),
                    )
                os.system(cmd)
                syslog.syslog(cmd)
                cmd = \
                    '{} -t filter {} FORWARD -d {} -p {} --dport {} -j ACCEPT'.format(iptables,
                        act, map_dict['ip'], proto, str(portmap[1]))
                os.system(cmd)
                syslog.syslog(cmd)
                cmd = \
                    '{} -t filter {} FORWARD -s {} -p {} --sport {} -j ACCEPT'.format(iptables,
                        act, map_dict['ip'], proto, str(portmap[1]))
                os.system(cmd)
                syslog.syslog(cmd)


host = mapping.get(domain)

if host is None:
    sys.exit(0)

if action == 'stopped' or action == 'reconnect':
    rules('-D', host)
    syslog.syslog('Removed all the iptables rules for ' + domain)

if action == 'start' or action == 'reconnect':
    rules('-I', host)
    syslog.syslog('Created all the iptables rules for ' + domain)

I also added this to my /etc/sysctl.conf

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
net.ipv6.conf.default.autoconf=0
net.ipv6.conf.default.accept_dad=0
net.ipv6.conf.default.accept_ra=0
net.ipv6.conf.default.accept_ra_defrtr=0
net.ipv6.conf.default.accept_ra_rtr_pref=0
net.ipv6.conf.default.accept_ra_pinfo=0
net.ipv6.conf.default.accept_source_route=0
net.ipv6.conf.default.accept_redirects=0
net.ipv6.conf.default.forwarding=0
net.ipv6.conf.all.autoconf=0
net.ipv6.conf.all.accept_dad=0
net.ipv6.conf.all.accept_ra=0
net.ipv6.conf.all.accept_ra_defrtr=0
net.ipv6.conf.all.accept_ra_rtr_pref=0
net.ipv6.conf.all.accept_ra_pinfo=0
net.ipv6.conf.all.accept_source_route=0
net.ipv6.conf.all.accept_redirects=0
net.ipv6.conf.all.forwarding=0
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.accept_redirects = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.log_martians = 0
net.ipv4.ip_local_port_range = 32768 61000
net.ipv4.icmp_echo_ignore_all = 0
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 2400
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_sack = 0

If you want you can add this too, but if you don’t want to you really need to add net.ipv4.ip_forward.

After editing /etc/sysctl.conf you need to issue

sysctl -p

To initialize and read the settings from /etc/sysctl.conf

Now you need to restart the libvirt service so it will initialize the hook, otherwise it will not call it.

libvirt Adds it’s own rules to ip tables, so if you want to do this manually you will have to make sure you add the correct rules for this to work.

For my setup the iptables was as follows, the rules will be automatically added/removed depending on the state of the machine whether it is running or not

# Generated by iptables-save v1.4.14 on Sat Dec 14 18:31:26 2013
*nat
:PREROUTING ACCEPT [126:7694]
:INPUT ACCEPT [126:7694]
:OUTPUT ACCEPT [223:17030]
:POSTROUTING ACCEPT [223:17030]
-A POSTROUTING -s 192.168.22.0/24 ! -d 192.168.22.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.22.0/24 ! -d 192.168.22.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.22.0/24 ! -d 192.168.22.0/24 -j MASQUERADE
COMMIT
# Completed on Sat Dec 14 18:31:26 2013
# Generated by iptables-save v1.4.14 on Sat Dec 14 18:31:26 2013
*mangle
:PREROUTING ACCEPT [12485:1642751]
:INPUT ACCEPT [12485:1642751]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [12074:3717410]
:POSTROUTING ACCEPT [12074:3717410]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Sat Dec 14 18:31:26 2013
# Generated by iptables-save v1.4.14 on Sat Dec 14 18:31:26 2013
*filter
:INPUT ACCEPT [12466:1639919]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [12074:3717410]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -d 192.168.22.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.22.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Sat Dec 14 18:31:26 2013

How to reset var folder permissions and ownership to their default in Debian

Thu, 28 Nov 2013 15:40:00 +0000

If by accident you execute a command that changes the permission or ownership of a bunch of folders or file in Debian, you are in a lot of trouble, usually the sollution is to reinstall the system, but as I was unwilling to reinstall thet system, I decided to find another way to do it, as I have finally set up my system as I want it to run, so we will look into a couple of methods of how to do it without reinstallation of the system.

Using virtual machine, reinstalling packages, or even generating a script from debian packages.

I accidently ran a chown command on the whole /var folder, and ended up with every file and folder in ownership of www-data.

According to the manual the var directory is:

The /var directory is mostly mounted as a separate filesystem under the root where in all the variable content like logs, spool files for printers, crontab,at jobs, mail, running process, lock files etc. Care has to be taken in planning this file system and maintenance as this can fill up pretty quickly and when the FileSystem is full can cause system and application operational issues.

I wanted to chown all the hidden files in the directory /var/www to www-data, I was in the directory /var/www when I ran the following command

chown -R www-data:www-data .*

And after running this, I noticed that my whole var folder was in ownership of www-data, so my quest to restore it to the original ownership and permissions started.

Imagine if it was rm instead of chown, I would have ended up deleting all the files in the var folder.

If you ever what to do something with the hidden files you should use the following approach, what I should have written should have been:

chown -R /var/www/.[^.]*

This command properly sets the hidden folders to the correct owners

So now I ended up with a var folder with a lot of messed up ownerships, so how to restore them.

There are multiple ways to do it, although the only way to completly restore them as they were would require you to deploy on a virtual machine, and copy the properties, that is the surest way so they are restored correctly

Method 1: Virtual Machine

Start installing you distribution on the virtual machine, and while you are waiting for the instalation to finish you need to prepeare some data needed for restoration of your machine.

This doesn’t have to happen on your machine, you can easily create a virtual machine on any computer and use the data to restore your own.

All the commands here are run as root.

We need a list of all your installed packages so they can be restored on the virtual machine, to get them, execute the following command

dpkg --get_selections > selections.txt

By now the machine should have finished installing, so let’s prepare some of the necessities, I use a pinning and a lot of repositores, including testing, unstable, stable, plus 3rd party repositories, so to do so we need to transfer the contents of /etc/apt to the virtual machine, you should know how to transfer the files to the virtual machine.

We also need to transfer the keys, because you can’t copy the keys we need to export them.

apt-key exportall > keys.gpg

And then transfer them to the virtual machine, and run the import for keys

apt-key add keys.gpg

If you are using multiarch you will have to add the appropriate architecture into the system, in my case I’m using also i386 libraries for skype, so I executed this on my virtual machine

dpkg --add-architecture i386

Now we need to install all the packages that are on our system, so let’s run the following commands on the virtual machine

apt-get update
dselect update
dpkg --set-selections < selections.txt

Now you can leave your virtual machine for a while, while it downloads all the packages and installs them.

After the installation has finished you can reboot your system, and after it boots, we need to extract all the information so we can restore it successfully.

On the virtual machine run the following command:

find /var -printf "%m:%u:%g:%p\n" > /tmp/var.permissions.txt

Now you transfer the generated file /tmp/var.permissions.txt to your machine, and you run the following script to restore everything as it was

#!/bin.bash
while IFS=":" read perms user group file; do 
chmod -R $perms $file > /dev/null 2>&1
chown -R $user:$group $file > /dev/null 2>&1
done < /tmp/var.permissions.txt

This is one way to restore them and it’s probably the best way to restore them

FreeBSD

If you have FreeBSD you can use mtree to copy over the permissions and restore them, the first line creates a list of the permissions and owners the second one restores them

mtree -c -k uname,gname,mode,time > /tmp/var.permissions.txt
mtree -U -f /tmp/var.permissions.txt

Method 2: From the debian packages

You can also restore permissions from the debian packages, but not the owners, as it doesn’t contain that information, it will restore them to how they are suppose to be when installing them the first time, but the permissions will be restored correctly.’

There are two ways to do this, you can restore them from the debian cache which is located in /var/cache/apt/archives/, but if you ever issued apt-get clean you will not have all the packages there, or you can just download all the packages and restore them like that, the script will be the same for both approaches.

So let’s create a new folder in opt, called restore

mkdir -p /opt/restore

I called my script create_permissions_script.sh, and I put it in /opt/restore, it is designed to work from the local directory the script is in.

This script reads all the packages, extracts the file list including permissions and owners, parses it and then stores it in a file so you can restore it.

#!/bin/bash
# /opt/restore/create_permissions_script.sh
ARCHIVE_DIR=`pwd`
PACKAGES=`ls $ARCHIVE_DIR/*.deb`
cd $ARCHIVE_DIR

cd /

function changePerms()
{
  CHOWN="/bin/chown"
  CHMOD="/bin/chmod"
  PERMS=`echo $1 | sed -e 's/--x/1/g' -e 's/-w-/2/g' -e 's/-wx/3/g' -e 's/r--/4/g' -e 's/r-x/5/g' -e 's/rw-/6/g' -e 's/rwx/7/g' -e 's/---/0/g'` 
  FTYPE=`echo ${PERMS:0:1}`
  PERMS=`echo ${PERMS:1}`
  OWN=`echo $2 | /usr/bin/tr '/' '.'`
  PATHNAME=$3
  PATHNAME=`echo ${PATHNAME:1}`
 
  if [[ "d" = "$FTYPE" ]]; then
    echo "$CHOWN -R $OWN $PATHNAME" >> "${ARCHIVE_DIR}/dir_perms.sh"
    echo "$CHMOD $PERMS $PATHNAME" >> "${ARCHIVE_DIR}/dir_perms.sh"
  else
    echo "$CHOWN $OWN $PATHNAME" >> "${ARCHIVE_DIR}/file_perms.sh"
    echo "$CHMOD $PERMS $PATHNAME" >> "${ARCHIVE_DIR}/file_perms.sh"
  fi;

  #fi;
}

for PACKAGE in $PACKAGES;
do
  if [ -d $PACKAGE ]; then
    continue;
  fi
  echo "Getting information for $PACKAGE"
  FILES=`/usr/bin/dpkg -c "${PACKAGE}"`
  for FILE in "$FILES";
  do
    echo "$FILE" | awk '{print $1"\t"$2"\t"$6}' | while read line;
    do
      changePerms $line
    done
  done
done

sort "${ARCHIVE_DIR}/dir_perms.sh" | uniq > "${ARCHIVE_DIR}/clean_dir_perms.sh"
sort "${ARCHIVE_DIR}/file_perms.sh" | uniq > "${ARCHIVE_DIR}/clean_file_perms.sh"

cat "${ARCHIVE_DIR}/clean_dir_perms.sh" "${ARCHIVE_DIR}/clean_file_perms.sh" > "${ARCHIVE_DIR}/restore_permissions.sh"
rm "${ARCHIVE_DIR}/clean_dir_perms.sh" "${ARCHIVE_DIR}/clean_file_perms.sh" "${ARCHIVE_DIR}/dir_perms.sh" "${ARCHIVE_DIR}/file_perms.sh"

This script will generate a new script called restore_permissions.sh, which contains all the neccesseary commands like chown and chmod for all the directories and files, executing this script will correct all the ownerships and permissions as they need to be after installation of the package in question

So let’s download the packages, or if you choose to restore them from the cache folder you can skip this step, just copy over this script to the cache folder and you are done, if you are like me and you ocasionally clean up the folder, you will have to download all the packages first

dpkg --get-selections \* | awk '{print $1}' | xargs -l1 apt-get download

Executing that line will download all the packages into the directory, as this will take a while, you can take a break.

After all the packages finish downloading, you need to run the script for generating the permissions and owners, this will also take a while to read and process all debian packages

./create_permissions_script.sh

After this is finished you will see that you have a script called restore_permissions.sh, and running it will restore the permissions

./restore_permissions.sh

Method 3: Reinstalling with aptitude

You can reinstall all the packages using aptitude, but this is a really slow process, but if you want to you can do like so

dpkg --get-selections \* | awk '{print $1}' | xargs -l1 aptitude reinstall

Automatic zone replication with BIND9 in Plesk 11.5 + Debian Wheezy + PHP

Sat, 19 Oct 2013 15:16:00 +0000

This tutorial is aimed at showing how to set up zone replication from a master to a slave server, without the help of the user input on Debian Wheezy with Plesk 11.5 and PHP

Let’s decide how to call the servers we are gonna use in the setup.

master

Main server, that hosts Plesk and BIND9 DNS server in Master mode

slave

Slave DNS server that hosts BIND9 DNS server in Slave mode

To be able to get a list of available domains on the main server we need mysql-client, on the DNS server, the reason for it is so we can get a list of domains from the primary server, so we can check if they exist or not so we can create them if necesseary.

So let’s install the client and some other requirements on the slave server, we can do so really easy by issuing the following command:

apt-get install mysql-client php5-cli php-pear php5-mysql

Once this is complete let’s go to the masters server and configure mysql to allow access from the slave server.

First thing first, let’s connect to the MySQL server on the master and create a user that will be only for viewing the domains records on the server.

If you need passwords you can use pwgen to generate random passwords, let’s generate 4 passwords with length 50.

$ pwgen 50 4
aed6Chu5AiphieJaeteirawei7yohrahZ2zaig8Eey9gePh4jo
ohw0sheiPhae1yae0eeghiesh8aevee5aineiyequae8ookaev
noh4icaigh2weiGaiph3sahDoeThaequ5HeGha2oeMeezee6re
Xaeb3uN6nib5cae2yotooz5quai5zohx8OhSah7caeshu3xeQu

Now let’s execute this SQL command to create a user for our case

CREATE USER 'dns'@'%' identified by 'noh4icaigh2weiGaiph3sahDoeThaequ5HeGha2oeMeezee6re';
GRANT USAGE ON *.* TO 'dns'@'%' with MAX_QUERIES_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_USER_CONNECTIONS 0;
GRANT SELECT ON psa.domains TO 'dns'@'%';
FLUSH PRIVILEGES;

These commands will enable you to query only the domains tables and nothing else, in case someone manages to find out the password they will only be able to query the domains we have. In our case this will allow a connection from anywhere if identified by the password, if you want to be more restrictive you can always specify the IP instead of % in the SQL.

Before we start we need to make sure we can add new zones in the system without restarting the server, to do so, we need to add to the options part of named.conf, the option to allow new zones, so we edit the file and add the following option in the config file.

allow-new-zones yes;

Now we can use rndc to create, update, read, delete the zones

Now, on to the next problem, we have the data, now we need to generate the zones when there is something new to add, we can do this by creating a script that is gonna run every 15 mins and query the dataset to check if there is new data that we need to input. We can do this simply in PHP, at the moment this is simplest took me 4 mins to write the whole code, so let’s take a look at the finished script:

The property parentDomainId, which I use to check which domains are TLD, at the moment, this script has been running for ~30 days and so far no issues at all.

#!/usr/bin/env php
<?php
define("PSA_USER", 'dns');
define("PSA_PASSWORD", 'noh4icaigh2weiGaiph3sahDoeThaequ5HeGha2oeMeezee6re');
define("PSA_DB", "psa");
define("PSA_HOST", "master");
define("BIND_VIEW", "slave");
define("PSA_SQL", "select name from domains where parentDomainId = 0");
define("LOCAL_DOMAIN_CACHE", "/opt/zones.cache");
define("NS_IPS", "10.0.0.1;");
define("TEMPLATE", "%%domain%% '{type slave; file \"slave/%%domain%%\"; masters { " . NS_IPS . " }; };'");

openlog("slave.configurator.php", LOG_PID | LOG_PERROR, LOG_LOCAL0);

$cache = array();

if (file_exists(LOCAL_DOMAIN_CACHE)) {
    $cache = json_decode(file_get_contents(LOCAL_DOMAIN_CACHE), true);
    if (is_null($cache)) {
        $cache = array();
    }
}

$link = mysqli_connect(PSA_HOST, PSA_USER, PSA_PASSWORD, PSA_DB) or die("Error " . mysqli_error($link));

if (!$link) {
    $err = mysqli_error($link);
    syslog(LOG_ERR, $err);
    die("Error: $err");
}

$result = $link->query(PSA_SQL);

$database_domains = array();

while ($domain = mysqli_fetch_assoc($result)) {
    array_push($database_domains, $domain['name']);
}

$process = array_diff($database_domains, $cache);
$full = array_unique(array_merge($database_domains, $process));

file_put_contents(LOCAL_DOMAIN_CACHE, json_encode($full));

$format = "Y-m-d H:i:s - ";

syslog(LOG_INFO,
        date($format, microtime(true)) . "Total available domains in cache: " . count($cache));
syslog(LOG_INFO,
        date($format, microtime(true)) . "Total available domains in database: " . count($database_domains));
syslog(LOG_INFO,
        date($format, microtime(true)) . "Difference elements between cache and database: " . count($process) . ", domains: " . (count($process) > 0 ? join(",",
                        $process) : "None available"));
syslog(LOG_INFO,
        date($format, microtime(true)) . "Total domains in system: " . count($full));

if (count($process) > 0) {
    foreach ($process as $domain) {
        echo date($format, microtime(true)) . "Processing domain: $domain\n";
        $template = str_replace("%%domain%%", $domain, TEMPLATE);
        syslog(LOG_INFO, "/usr/sbin/rndc addzone $template");
        exec("/usr/sbin/rndc addzone $template");
    }
    exec("/usr/sbin/rndc reload");
} else {
    syslog(LOG_INFO, date($format, microtime(true)) . "Nothing to process.");
}

NS_IPS, is a ; separated list of all your nameservers, you should put your IP of your master nameserver here, in the demo script I put 10.0.0.1, but you should replace it with your own master nameserver, the IP should always have a ; at the end of the IP in the script it was 10.0.0.1;

If /usr/sbin/rndc is not the correct path for rndc, you should change the script accordingly;

Now you can add it to crontab this script I saved mine under /opt/slave.configurator.php, and my crontab entry is like so, I run this script under root.

*/15 * * * * /opt/slave.configurator.php

Just don’t forget to run:

chmod +x /opt/slave.configurator.php

Generating DomainKeys for Mail

Thu, 10 Oct 2013 23:47:00 +0000

We need to generate DK (DomainKeys) to validate that we are the original senders and not someone impersonating us, this tutorial is meant to explain how generate the keys

To generate a DomainKeys record we can do simply from the command line.

openssl genrsa -out default 1024
openssl  rsa -in default -out txt -pubout -outform PEM

This will create two files that will contain the private and the public key that we can put in the DNS zone record, let’s check the files what they contain.

cat default

-----BEGIN RSA PRIVATE KEY-----
MIICXAIBAAKBgQCvcvkiUaVlHZNDo7grpSh+SLc8L5yZMg34uysFdc8HNWB/bbx9
lEMUj2/n/1qlj+AYcisMIpCGGK+LrrGig65x7OH31mpxWjj8zfKNtGzSjdBzgbL+
kMZi21VpoJ8BP/R9oGrJf4woZ/arZUmlzKpTsNqw7RCvNJ55OCAi2jcJWQIDAQAB
AoGBAJfzw8HtZInGq5yRVxi12fRFli0SL1ae+2rI7+GyvrNHj2PN7sn0doSAFjOf
/SoXCcciWhYQeYsqJh+cFUzjL1kc7NqqI8rX36L49G+ur87AJFDYsz851OCly9JN
9rAe36Dv7BToE+hC0cqT7u5bNjKEimNNHsdprQlRm6RiFbABAkEA1Oalhe5GGjsy
ICO7X5JCyF7QI0yjLPY4cP9hHvgS+WPT5/IciQIUKmV2T6+TS48nFgAU5euFjepZ
Bzq4KzhHmQJBANL3bTMoVB0yh8fckN12KFx34DdOCclBqu0vNXTNcEoHspAJhaa5
DlQuZCUursPDwibBv/exWx4BgfFMOQJe58ECQCP/T5Nio1XCFoqaoA7bwxDv/w9I
4Po0M3zfoUNEPKkQOP8pz6tWv6Qffa6hiC0pajltEBuEBBPnwN/ZDNS58lkCQHMc
HFJQi+zOeHXd7JFZ+lXR9t5WT1Kn6Qq3upQ70CwknRKoj2tUB/R4x53eJe+dLZ+W
EhelhxENQ4iUzXp0rEECQEJIsNDQIeDcOyMKy1TOKFwG2TFvvIqDlM7+2NKucmcy
Re2q6DPYpksiJVY/2EAtYqsBAEztVb38bakbY5EUmp0=
-----END RSA PRIVATE KEY-----

And let’s check the txt file

cat txt

-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvcvkiUaVlHZNDo7grpSh+SLc8
L5yZMg34uysFdc8HNWB/bbx9lEMUj2/n/1qlj+AYcisMIpCGGK+LrrGig65x7OH3
1mpxWjj8zfKNtGzSjdBzgbL+kMZi21VpoJ8BP/R9oGrJf4woZ/arZUmlzKpTsNqw
7RCvNJ55OCAi2jcJWQIDAQAB
-----END PUBLIC KEY-----

From this you can see that we have a public and a private key, the private one we keep for us, if someone has the private key, they can sign as us.

To convert the txt file to a format suitable for DNS record we need to copy the key, and delete all the new lines

MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvcvkiUaVlHZNDo7grpSh+SLc8L5yZMg34uysFdc8HNWB/bbx9lEMUj2/n/1qlj+AYcisMIpCGGK+LrrGig65x7OH31mpxWjj8zfKNtGzSjdBzgbL+kMZi21VpoJ8BP/R9oGrJf4woZ/arZUmlzKpTsNqw7RCvNJ55OCAi2jcJWQIDAQAB

So we just prepend the value p= and append ; and we have our key that we can put in a DNS zone

So let’s see the complete record now:

_domainkey.example.com.	         IN TXT	 "o=-"
default._domainkey.example.com.  IN TXT	 "p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvcvkiUaVlHZNDo7grpSh+SLc8L5yZMg34uysFdc8HNWB/bbx9lEMUj2/n/1qlj+AYcisMIpCGGK+LrrGig65x7OH31mpxWjj8zfKNtGzSjdBzgbL+kMZi21VpoJ8BP/R9oGrJf4woZ/arZUmlzKpTsNqw7RCvNJ55OCAi2jcJWQIDAQAB;"

Setting up SPF + DK + DKIM with Postfix in Plesk 11.5 on Debian Wheezy

Thu, 10 Oct 2013 12:55:00 +0000

Shows up a detailed process on how to set up SPF + DK + DKIM with Postfix in Plesk 11.5 on Debian Wheezy, step by step, and how to test to make sure everything is working correctly

I leased a dedicated server from Hetzner, and I got the Plesk option, for administration, so I don’t have to bother with administration, but turns out I’m not so lucky, I’ve ran into a lot of issues with using Plesk, so I had to do my own fixes.

So let’s take a look at how we can integrate SPF + DK + DKIM with Postfix in Plesk 11.5 on Debian Wheezy.

First things, first, if you are using QMail switch to Postfix, to install Postfix you can either use the GUI, or you can do it from a console.

Here is how to do it from the console.

/usr/local/psa/admin/sbin/autoinstaller --select-release-current --install-component postfix

SPF

Let’s open the DNS Template, you will see there that, there is an entry for SPF

v=spf1 +a +mx -all

This means the SPF is enabled on our domain.

Let’s modify it a little bit to be better, if you are gonna host multiple domains from your server, then you should probably modify it too.

v=spf1 +a +mx +ip4:<ip.mail> ?all

<ip.mail> - Is the IP of the mail server that is responsible for sending the mails, it is automatically filled in when you apply the zones

After you do this modification you should apply it

So now my configuration looks like this:

Now let’s check with dig if the SPF is OK.

dig myserverplace.de TXT @ns1.myserverplace.de

You will see in the Answer section I have the following entry

myserverplace.de.	600	IN	TXT	"v=spf1 +a +mx +ip4:144.76.163.46 ?all"

So everything is ok, next onto DomainKeys

DomainKeys

First let’s activate DomainKeys, take a look at the screenshot, and compare my options with yours.

Ok, now that this has been enabled, let’s go and enable for the domain in question, if it was already checked, uncheck it press OK, and then check it again and press OK, this is so it will regenerate the DomainKeys data in the DNS zone, as I’ve had some problems with the data not present in the DNS zone file

OK, now let’s see if the correct data is there, usually it takes a long time for DNS to propagate between 24-48h, there is a simple way to test if the data is there, by querying the Nameserver that hosts your DNS zone, in my case I host my own Nameserver

dig _domainkey.myserverplace.de TXT @ns1.myserverplace.de

You will see in the Answer section I have the following entry

_domainkey.myserverplace.de. 600 IN	TXT	"o=-"

Now let’s see if the DomainKey is there too

dig default._domainkey.myserverplace.de TXT @ns1.myserverplace.de

In the Answer section you should see something like

default._domainkey.myserverplace.de. 600 IN TXT	"p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDAruBNqdsSCKBLwMrFNNKH8z0e7zmlAic7iRoJsDDJK3IlnW8j6G/T6a93m+jqYc6R38MBAZbeSv2LQJ0SepJEsr4Iqk41WFXPBKnyXReO1RXPW5/YnRe6dpJMEqsmPpl2TjInY7ve/6VCiVDOHn9RRrdB+x7CGeK2crgqSZVlFwIDAQAB\;"

As you can see everything is in place now for DomainKeys to work, now let’s continue on to DKIM

DKIM (DomainKeys Identified Mail)

As always lets update the system first

aptitude update
aptitude safe-upgrade

Now we need to install the DKIM filter, or as it’s called now OpenDKIM, for full specification take a look at their site.

aptitude install opendkim opendkim-tools

Now we need to create the necessary folders so OpenDKIM can work proplery

mkdir -pv /etc/opendkim/keys
chown -Rv opendkim:opendkim /etc/opendkim
chmod go-rwx /etc/opendkim/*

This will create the directory where we will hold the keys for OpenDKIM, after this step let’s take a look at how the process will look like, so we can create a script to automate this.

mkdir -p /etc/opendkim/keys/myserverplace.de
cd /etc/opendkim/keys/myserverplace.de
opendkim-genkey -d myserverplace.de -s mail
chown -Rv opendkim:opendkim /etc/opendkim/keys/myserverplace.de
chmod -v u=rw,go-rwx *

This easily understandble, what happens here.

Now you have two files /etc/opendkim/keys/myserverplace.de

ls -lah /etc/opendkim/keys/myserverplace.de
total 16K
drwxr-xr-x 2 opendkim opendkim 4.0K Oct  9 18:43 .
drwxr-xr-x 5 opendkim opendkim 4.0K Oct  9 19:39 ..
-rw------- 1 opendkim opendkim  887 Oct  9 18:43 mail.private
-rw------- 1 opendkim opendkim  303 Oct  9 18:43 mail.txt

/etc/opendkim/keys/myserverplace.de/mail.private

contains the RSA PRIVATE KEY

/etc/opendkim/keys/myserverplace.de/mail.txt

Contains the record you need to add to your DNS zone

Next set is to setup the key tables, signing tables, and trusted hosts

First let’s prepare the files

touch /etc/opendkim/KeyTable
touch /etc/opendkim/SigningTable
touch /etc/opendkim/TrustedHosts

/etc/opendkim/TrustedHosts

needs to contain some data before we continue, so add the following information to this file, and adjust accordingly
```
127.0.0.1
localhost
144.76.163.46
144.76.163.57
ns1.myserverplace.de
ns2.myserverplace.de
myserverplace.de
```

So let’s see what does what:

/etc/opendkim/KeyTable

KeyID Domain:Selector:PathToPrivateKey

/etc/opendkim/SigningTable

The filter used is programmed to read the table by looking for matched domain

/etc/opendkim/TrustedHosts

It will list the top trusted hosts as you desire

Those three files contain all the necessary information for the signing to work.

So in my case for my domain I do.

echo "myserverplace.de myserverplace.de:mail:/etc/opendkim/keys/myserverplace.de/mail.private" >> /etc/opendkim/KeyTable
echo "*@myserverplace.de myserverplace.de" >> /etc/opendkim/SigningTable
echo "myserverplace.de" >> /etc/opendkim/TrustedHosts
echo "mail.myserverplace.de" >> /etc/opendkim/TrustedHosts

So let’s put all this together in a script so we don’t have to do it all the time

#!/bin/bash
# /opt/generatedkim.sh
die () {
    echo >&2 "$@"
    exit 1
}

[ "$#" -eq 1 ] || die "1 argument required, $# provided, domain required, ex: ./script example.com"

cwd=`pwd`
opendkim="/etc/opendkim"
location="$opendkim/keys/$1"
[ -d "$location" ] && die "There is already a directory in the folder, delete the folder if you want to create a new one"

mkdir -p "$location"
cd "$location"
opendkim-genkey -d $1 -s mail
chown opendkim:opendkim *
chown opendkim:opendkim "$location"
chmod u=rw,go-rwx *
echo "$1 $1:mail:$location/mail.private" >> "$opendkim/KeyTable"
echo "*@$1 $1" >> "$opendkim/SigningTable"
echo "$1" >> "$opendkim/TrustedHosts"
echo "mail.$1" >> "$opendkim/TrustedHosts"
echo
echo "Put this in the DNS ZONE for domain: $1"
echo
cat "$location/mail.txt"
echo
cd "$cwd"

So if we run the script, we should get output like this, and this is the data we need to put in the DNS zone.

/opt/generatedkim.sh test.de

Put this in the DNS ZONE for domain: test.de

mail._domainkey IN TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDPzE0GmvFwAQsgcFzopy4zMNWUbL6JM5XIyjBy3bUnANI5axeb/Lw/GBjUoSFLEiO80Tt8m3A5YrBKcodRQQURYiW6/YtElhLupHyfcxQhfNLU4z9JUOJKPjcpMZCj0Xv873QgVOl+7U605JdBHSPOx4ybBZwDq68cw9YFYRPmEwIDAQAB" ; ----- DKIM key mail for test.de

Unfortunatly I don’t have time to create a script to do this automatically, you can always insert a record in MySQL database so it’s in the ZONE and you can regenerate the DNS Zone from the command line, and I won’t be having a lot of domains, so I can add this entry manually to a domain I want DKIM enabled

Let’s open the domain and go to DNS Settings, and you can click ** Add Resource **

You popuplate the following data in the inputboxes

Record type

TXT

Domain name

mail._domainkey

TXT Record

In the text record you copy a part of the contents from the file /etc/opendkim/keys/myserverplace.de/mail.txt, it data should start from v=DKIM1; k=rsa; to the end, without the quotes as you can see it’s in quotes.

In the example above for domain test.de you add only the following contents in the input box

v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDPzE0GmvFwAQsgcFzopy4zMNWUbL6JM5XIyjBy3bUnANI5axeb/Lw/GBjUoSFLEiO80Tt8m3A5YrBKcodRQQURYiW6/YtElhLupHyfcxQhfNLU4z9JUOJKPjcpMZCj0Xv873QgVOl+7U605JdBHSPOx4ybBZwDq68cw9YFYRPmEwIDAQAB

Well that’s it for this, now let’s check with dig if the record is there

dig mail._domainkey.myserverplace.de TXT @ns1.myserverplace.de

You will see in the Answer section I have the following entry

mail._domainkey.myserverplace.de. 600 IN TXT	"v=DKIM1\; k=rsa\; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDMziMcgPTWK0kSUKxrgHHzEiWxNkZ2/M0Ugyr/8H9WtoCsJUM+Bc1C9VwqJ6yjTidecDrX7aL0lFZ9Mylku/wtSiPw6KLxMg2LG2vrMzlPTB2lmJNmg/EOu3KPC8BtAuOhXfwVH/ttQbzdKJKWqiCJn7jhF5oqEKnOCORxOQXKIwIDAQAB"

Well everything is setup up at least from the DNS side, now we need to configure Postfix to use this data and sign the emails.

You can also use the following URLs to check the validity of your key

In the selector fields try with both mail, and default, you shold be getting valid results

OpenDKIM

We need to edit the configuration file to configure DKIM, open /etc/opendkim.conf with your favorite editor and add the following lines to the end of the file

# Enable Logging
Syslog yes
SyslogSuccess yes
LogWhy yes

# User mask
UMask 002

# Always oversign From (sign using actual From and a null From to prevent malicious signatures header fields (From and/or others) between the signer and the verifier)
OversignHeaders From

# Our KeyTable and SigningTable
KeyTable refile:/etc/opendkim/KeyTable
SigningTable refile:/etc/opendkim/SigningTable

# Trusted Hosts
ExternalIgnoreList /etc/opendkim/TrustedHosts
InternalHosts /etc/opendkim/TrustedHosts

# Hashing Algorithm
SignatureAlgorithm rsa-sha256

# Auto restart when the failure occurs. CAUTION: This may cause a tight fork loops
AutoRestart Yes

# Set the user and group to opendkim user
UserID opendkim:opendkim

# Specify the working socket
Socket inet:8891@localhost

That’s it for OpenDKIM, now we should restart the service

service opendkim restart

Postfix

Now let’s see what we need to do to configure Postfix to use OpenDKIM.

Execute the following command to see the milters configured

cat /etc/postfix/main.cf | grep "milters"
smtpd_milters = , inet:127.0.0.1:12768
non_smtpd_milters = , inet:127.0.0.1:12768

You can see that we have additional milters we need to put, this one is from the process psa-pc-remote, and it’s part of Plesk

Open /etc/postfix/main.cf with your favorite editor, and add the following to the end of the file

# OpenDKIM
milter_default_action = accept
milter_protocol = 6
smtpd_milters = , inet:127.0.0.1:8891, inet:127.0.0.1:12768
non_smtpd_milters = $smtpd_milters

As you can see we added the OpenDKIM milter too, and milter_protocol is set to 6, this is important, if it’s not set to 6, the psa-pc-remote process will segfault like so,

psa-pc-remote[18523]: segfault at 0 ip 00007fa5be18c034 sp 00007fa5bccffd30 error 4 in libc-2.13.so[7fa5be123000+180000]

And your messages won’t be signed with DomainKey, only with DKIM

service postfix restart

Testing

There is an easy way to test if everything is correct, create an email account if you haven’t already and send a test mail to the following recepients, and the results are cut down because the text is too big

check-auth@verifier.port25.com

==========================================================
Summary of Results
==========================================================
SPF check:          pass
DomainKeys check:   pass
DKIM check:         pass
Sender-ID check:    pass
SpamAssassin check: ham

AAAA3QcKCQwA@appmaildev.com

============================================================
SPF result: Pass
============================================================
Domain: myserverplace.de
IP: 144.76.163.46

SPF Record: myserverplace.de
        IN TXT = "v=spf1 +a +mx 144.76.163.46 ?all"

============================================================
DomainKey result: pass
============================================================
Signed by: admin@myserverplace.de

PublicKey: default._domainkey.myserverplace.de
        IN TXT = "p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDAruBNqdsSCKBLwMrFNNKH8z0e7zmlAic7iRoJsDDJK3IlnW8j6G/T6a93m+jqYc6R38MBAZbeSv2LQJ0SepJEsr4Iqk41WFXPBKnyXReO1RXPW5/YnRe6dpJMEqsmPpl2TjInY7ve/6VCiVDOHn9RRrdB+x7CGeK2crgqSZVlFwIDAQAB;"

============================================================
DKIM result: pass
============================================================
Signed by: admin@myserverplace.de
Expected Body Hash: frcCV1k9oG9oKj3dpUqdJg1PxRT2RSN/XKdLCPjaYaY=

PublicKey: mail._domainkey.myserverplace.de
        IN TXT = "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDMziMcgPTWK0kSUKxrgHHzEiWxNkZ2/M0Ugyr/8H9WtoCsJUM+Bc1C9VwqJ6yjTidecDrX7aL0lFZ9Mylku/wtSiPw6KLxMg2LG2vrMzlPTB2lmJNmg/EOu3KPC8BtAuOhXfwVH/ttQbzdKJKWqiCJn7jhF5oqEKnOCORxOQXKIwIDAQAB;"

Logs

You can check the following locations to see if there are errors

/var/log/mail.err
/var/log/mail.warn
/var/log/mail.info
/var/log/syslog

Note

Make sure you enable testing mode for DKIM if you plan to test, you can also cut down the EXPIRY time so the results propagate faster, so to enable testing mode set the key _domainkey to t=y; o=-

References

How to define what MTA is used in Parallels Plesk Panel and how to switch from Qmail to Postfix and back?

Debian Backup with Tartarus

Thu, 03 Oct 2013 19:04:00 +0000

How to backup a Debian/Ubuntu server with Tartarus, and why is this usefull for us

Why do you need this, if you are like me and like to experiment a lot, you will end up breaking your server a lot of times, so this will enable you to restore your server fast and easy.

To be able to use Tartarus first we need to install it, so let’s issue the following commands.

wget -O /etc/apt/sources.list.d/wertarbyte.list http://wertarbyte.de/apt/wertarbyte-apt.list
wget -O - http://wertarbyte.de/apt/software-key.gpg | apt-key add -
apt-get update
apt-get install tartarus tar bzip2 lvm2 gnupg curl

This will add and install the application on your server and some extra applications needed. Now we need to create the required directories for tartarus.

mkdir /etc/tartarus
mkdir -p /var/spool/tartarus/timestamps

Let’s create a passphrase file, we can do this by executing the command

pwgen -s 50 1 > /etc/tartarus/backup.sec

That command will create a file that contains a random generated password, you can just do

cat /etc/tartarus/backup.sec
JvXqwrouhnV1xaGS563N1ivvmgevUaCVLX13kHkYUmJH6gCiql

So you can see that we have gotten a random secure password. Next thing, we need to setup a general configuration, that all the other configurations are gonna extend from.

touch /etc/tartarus/generic.inc

Now open it up in your favorite editor, and adjust your settings to match.

STORAGE_FTP_SSL_INSECURE="yes"
STORAGE_METHOD="FTP"
STORAGE_FTP_SERVER="<FTP SERVER>"
STORAGE_FTP_USER="<FTP USER>"
STORAGE_FTP_PASSWORD="<PUT YOUR FTP PASSWORD HERE>"
STORAGE_FTP_USE_SFTP="yes"

COMPRESSION_METHOD="bzip2"
STAY_IN_FILESYSTEM="yes"

ENCRYPT_SYMMETRICALLY="yes"
ENCRYPT_PASSPHRASE_FILE="/etc/tartarus/backup.sec"

TARTARUS_POST_PROCESS_HOOK() {
    echo -n "$STORAGE_FTP_PASSWORD" | /usr/sbin/charon.ftp \
    --host "$STORAGE_FTP_SERVER" \
    --user "$STORAGE_FTP_USER" --readpassword \
    --maxage 7 \
    --dir "$STORAGE_FTP_DIR" --profile "$NAME"
}

TARTARUS_DEBUG_HOOK() {
    echo $DEBUGMSG | logger
}

So this is the generic file from which everyone will extend. Now we have everything, all the variables are set, and we an create our first backup script. Let’s create a complete root backup.

touch /etc/tartarus/root.conf

After opening in your favorite editor add the following lines

source /etc/tartarus/generic.inc
NAME="root"
DIRECTORY="/"
EXCLUDE="/tmp/ /proc/ /sys/ /var/tmp/ /var/run/ /var/lib/mysql/"
CREATE_LVM_SNAPSHOT="no"
INCREMENTAL_STACKING="yes"
INCREMENTAL_TIMESTAMP_FILE="/var/spool/tartarus/timestamps/root"

You can see we excluded some of the directories for backup. Let’s explain some of the options, for a full list of options you can check them at Tartarus

INCREMENTAL_TIMESTAMP_FILE

Everytime a full backup is successfully completed, Tartarus willtouch the file specified here as a reference point for future incremental backups.

INCREMENTAL_STACKING

With this option enabled, Tartarus will also update the flagfile after completing a successfull partial (differential/incremental) backup run. By that, incremental backups are “stacked” on each other instead of being based on the most recent full backup.

Now let’s create a backup script that we gonna put into cron to run everyday.

touch /usr/local/sbin/tartarus-backup.sh
chmod u+x /usr/local/sbin/tartarus-backup.sh

And with your editor add the following contents

#!/bin/sh
# /usr/local/sbin/tartarus-backup.sh
# Run all backup profile found in /etc/tartarus/ and pass
# command line arguments on to tartarus (e.g. -i)
for profile in /etc/tartarus/*.conf; do
 /usr/sbin/tartarus $* "$profile"
done

So in crontab we add the following, so it will do a full backup every sunday, and from monday to saturday incremental backup

crontab -e

And copy paste the following entries

0 1 * * mon-sat /usr/local/sbin/tartarus-backup.sh -i
0 1 * * sun /usr/local/sbin/tartarus-backup.sh

When you do a

crontab -l

You should be able to see them in the output OK, so beacuse this is the first time, let’s run the backup script so it creates full backups

/usr/local/sbin/tartarus-backup.sh

Restoring a backup

Since tartarus is based on linux utilities, restoring a backup from a rescue system is extremely easy, in case of incremental stacking you will need to extract all the files in the order they were created, you still use the same command to extract them.

mkdir /mnt/restore
curl ftp://USER:PASS@YOURSERVER/DIR/FILE | gpg --decrypt | tar xpvj -C /mnt/restore

Your backup is restored in the directory /mnt/restore

Activate maintenance mode manually in WordPress

Mon, 30 Sep 2013 12:23:00 +0000

In the previous post I wrote about how to transfer my own WordPress blog, to a static generated site using Wintersmith, but now, I need to take down the blog, so first thing, first, we need to create backup of the database and content I have uploaded on the system. So lets see how we can do it.

I needed to put my WordPress into maintenance mode, so I can create a backup of my database, as I don’t want to lose all the information it has, like users, posts, comments, content, and so on.

The first step was to put the blog into maintenance mode, to do so, we can do fairly easily, the only thing we need to do is create a file called .maintenance in the root of the blog folder.

To do so we can execute the following command:

echo "<?php \$upgrading = time(); ?>" > .maintenance

This will create the file with the necesseary contents for the blog to be in maintenance mode. So now your site should be in maintenance mode for a while or until you remove the file.

The function uses the following sum to work out how long to stay in maintenance mode for

The time now = The time specified – 10 minutes

So in the above example when we are using the time() function the sum will always be within 10 minutes.

If you want you can also just supply the time in unix timestamp for when you want the in the future, let’s say in Sun, 20 Oct 2030 10:30:00 GMT so in that case you can create the file like:

echo "<?php \$upgrading = 1918723200; ?>" > .maintenance

If you want to stop maintenance mode, it’s really simple, just delete the file and the maintenance mode is over, and the blog will continue functioning‎ as normal

Migration from Wordpress to Wintersmith static site generator

Sun, 29 Sep 2013 19:30:00 +0000

Well my blog was on Wordpress, and I was happy, so why did I move away?

Constant maintenance, requires a lot of time, as it’s constantly being targeted by hackers
Slow, even with caching plugins, I didn’t want to put a really powerfull machine just to run a blog page
Server maintenance
Didn’t use all the functionality in the system, don’t really need all that functionality

So I got tired of managing all this stuff, so I decide it is time to find something that won’t waste so much of my time and let me use it when I need, and not worry about if I’m gonna get hacked, or if my server is slow.

Let’s see some of the pros and cons of running a static generated site.

Pros:

Extremely fast
Don’t need to worry, if someone is gonna find an exploit for my site, even if I mess up, it’s still html and javascript code, there is nothing they can do on the server, to change content and stuff like that
You can write your posts in Markdown format

Cons:

You need your computer to publish a new post, you can’t do it from the beach or the phone if you want to do it. (Though there is way of how to do this automatically, to accomplish simillar thing, like creating a markdown document, and pushing it to git, while the server periodically regenerates the site if there are new changes)
You need to regenerate the build and deploy it
Needed to learn Jade, there are other templating plugins available, but in the end I settled with the default one, as it wasn’t really difficult to learn, took about one hour to be able to use it

Choosing Static Site Generator

These are some of the generators that I checked to see if I can use them, after looking through the web to see what other people are using I narrowed the selection down to these items:

I played a bit with all of them and in the end I settled with Wintersmith, the reason is, that is really easy to use, and extendable.

Preparation

So firstly I needed to prepare the environment for Wintersmith.

A lot of people install node from repositories, but I prefer to use Node Version Manager, it gives me more control over which version I use, for details about how to install visit this page.

To get started, you will need to install Wintersmith. Getting Wintersmith installed is fairly easy when using npm. You simply need to type the following:

npm install -g wintersmith

One of the features that was greatly improved with some of the newer versions of Wintersmith is the ability to generate a new site from the command line utility. This saves a great deal of time in the early creation process. The command can be utlized as follows:

wintersmith new <project_name>

In the following example, I ran the following command with project name of blog. This generated a basic skeleton for me to use in building this blog.

Looking at the skeleton of the blog, it is fairly simple to use, and configure.

Modding

I needed some data added to the site, so I can safely migrate from my old Wordpress blog to my new static generated one.

Helper

I needed a function that will return all articles so I can manipulate the generation process, and use it to generate the redirects, sitemap and feeds

var _ = require('underscore'), util = require('util');

module.exports = function(env, callback) {

    var defaults = {
        articles: 'articles'
    };

    var options = _.extend(env.config.helpers || {
    }, defaults);

    /**
     * Get all available articles
     * @param {Object} contents
     *
     * @returns {Array} A list of available articles sorted by descending date
     */
    var getArticles = function(contents) {
        return contents[options.articles]._.directories.map(function(item) {
            return item.index;
        }).sort(function(a, b) {
            return b.date - a.date;
        });
    };

    // add the article helper to the environment so we can use it later
    env.helpers.getArticles = getArticles;

    return callback();

};

Robots

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.

- // robots.txt
| User-agent: *
| Disallow: #{locals.robotsDisallowed}
| Sitemap: #{locals.url}/sitemap.xml

In the main config file under locals we create robotsDisallowed property which holds what is not allowed, in my case it’s empty.

So the generated file looks like this:

User-agent: *
Disallow:
Sitemap: http://blog.matoski.com/sitemap.xml

Sitemap

A site map (or sitemap) is a list of pages of a web site accessible to crawlers or users. It can be either a document in any form used as a planning tool for Web design, or a Web page that lists the pages on a Web site, typically organized in hierarchical fashion. There are two popular versions of a site map. An XML Sitemap is a structured format that a user doesn’t need to see, but it tells the search engine about the pages in your site, their relative importance to each other, and how often they are updated. HTML sitemaps are designed for the user to help them find content on the page, and don’t need to include each and every subpage. This helps visitors and search engine bots find pages on the site.

doctype xml
urlset(
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance",
  xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd",
  xmlns="http://www.sitemaps.org/schemas/sitemap/0.9")
  - var buildDate = new Date().toISOString()
  url
    loc =locals.url
    lastmod= buildDate
    changefreq daily
    priority 1.0
  url
    loc= locals.url + "/archive.html"
    lastmod= buildDate
    changefreq daily
    priority 1.0
  - var articles = _.chain(contents.articles._.directories).map(function(item) {
  -   return item.index
  - }).compact().filter(function(article) {
  -   return article.metadata.ignored !== true
  - }).sortBy(function(item) {
  -   return -item.date
  - }).value()
  for article in articles
    - var permalink = locals.url + article.url
    url
      loc= permalink
      lastmod= article.date.toISOString()
      changefreq daily
      priority 0.8

.htaccess

A .htaccess (hypertext access) file is a directory-level configuration file supported by several web servers, that allows for decentralized management of web server configuration. They are placed inside the web tree, and are able to override a subset of the server’s global configuration for the directory that they are in, and all sub-directories.

The reason for this file is so we can provide redirects to the new articles in the old system, this will work on Apache only, for a nginx configuration you can see it bellow

- var articles = env.helpers.getArticles(contents);
| RewriteEngine On
| RewriteBase /　#{"\n"}
for article in articles
    if article.metadata.oldurl
        | Redirect 301 #{article.metadata.oldurl} #{article.url} #{"\n"}

nginx.conf

This file generates the redirects for the old articles in the system to the new system, but you will have to put them manually inside in the configuration file for the host

- var articles = env.helpers.getArticles(contents);
for article in articles
    if article.metadata.oldurl
        | location #{article.metadata.oldurl}
        |   rewrite ^(.*)$ #{article.url} redirect;
        | } #{"\n"}

Google Analytics Tracking

At the end of layout.jade, I added this bellow the footer, this adds google tracking to my web page

     script(type='text/javascript').
        var _gaq = _gaq || [];
        _gaq.push(['_setAccount', '#{locals.googleTrackingCode}']);
        _gaq.push(['_trackPageview']);
        (function() {
          var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
          ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
          var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
        })();

The locals.googleTrackingCode contains my tracking code, and it’s set in the config.json in the root of the directory.

So that’s it the blog is ready to go.

Next thing to do is to style this up, I’m thinking what CSS framework to use, I’m thinking about using Foundation, so should be fun.

So total in all it took about 2.5 hours to transfer my old blog, to the new static system, not much if you compare the speed you get and stability with the static site builder.

Building lftp on Windows

Wed, 25 Sep 2013 20:50:00 +0000

One of the best FTP clients I have ever seen is lftp, and what is even best it’s open source on Linux, if you want to use it on Windows you will have to compile it with Cygwin.

Why would someone use lftp, when there are alternatives like FileZilla, and a lot of other ones. Yes FileZilla is good and it’s free, but some of the options that are available with lftp are really good and helpful. And I know of a lot of people that use lftp for server backups on Windows Server and Linux machines to do recursive backups and mirroring.

For a complete feature list you can see this page.

Some interesting information about this application can be found on this page, there are various examples of how to use it on their page.

If you want to learn how to compile it on Windows, read bellow, or if you are just interested in the compiled version you can download it bellow, the following builds are built on Cygwin x86x64, also bundled in the package is SSH and BASH. If you have Cygwin installation you can just unpack this file in the root, and you are set.

At the bottom of the post you will find a script that was used to build these packages for anyone who want

So first things first, download the latest version of lftp, and unpack it somewhere where you can access it, in my case I uncompressed it in my home directory, so at the time of writing the latest version is 4.4.9.

Here are some of the perquisites for compiling the application successfully on Cygwin.

In short you need the following packages for you to be able to compile lftp, for a more complete list of packages that I use to compile you can find them at the end of the post.

bison, autoconf, gcc, gcc-c++, make, pkg-config, gnutls, gnutls-devel

In the main directory we run:

./configure
make

So after compilation let’s check what support do we have in lftp:

lftp.exe --version

It shows us that we are using: Readline 6.2, Expat 2.1.0, GnuTLS 3.2.0, libiconv 1.14, zlib 1.2.8

ldd.exe lftp.exe | grep --invert-match Windows

This shows you all the dependencies for the particular executable, in our case it shows the following.

ldd.exe lftp.exe | grep --invert-match Windows
        cygwin1.dll => /usr/bin/cygwin1.dll (0x180040000)
        cygexpat-1.dll => /usr/bin/cygexpat-1.dll (0x3ff980000)
        cyggnutls-28.dll => /usr/bin/cyggnutls-28.dll (0x3ff590000)
        cyggmp-10.dll => /usr/bin/cyggmp-10.dll (0x3ff690000)
        cyghogweed-2.dll => /usr/bin/cyghogweed-2.dll (0x3ff4d0000)
        cygnettle-4.dll => /usr/bin/cygnettle-4.dll (0x3ff0e0000)
        cygiconv-2.dll => /usr/bin/cygiconv-2.dll (0x3ff3c0000)
        cygintl-8.dll => /usr/bin/cygintl-8.dll (0x3ff360000)
        cygp11-kit-0.dll => /usr/bin/cygp11-kit-0.dll (0x3ff0a0000)
        cygffi-6.dll => /usr/bin/cygffi-6.dll (0x3ff970000)
        cygtasn1-6.dll => /usr/bin/cygtasn1-6.dll (0x3feba0000)
        cygz.dll => /usr/bin/cygz.dll (0x3feb40000)
        cygreadline7.dll => /usr/bin/cygreadline7.dll (0x3fede0000)
        cygncursesw-10.dll => /usr/bin/cygncursesw-10.dll (0x3ff120000)
        cyggcc_s-seh-1.dll => /usr/bin/cyggcc_s-seh-1.dll (0x3ff850000)

If you want to compile with OpenSSL support do the following, because by default it doesn’t compile with OpenSSL support, instead it uses GnuTLS.

./configure --without-gnutls --with-openssl
make

So after compilation let’s check what support do we have in lftp:

lftp.exe --version

It shows us that we are using: Readline 6.2, Expat 2.1.0, OpenSSL 1.0.1e 11 Feb 2013, libiconv 1.14, zlib 1.2.8

ldd.exe lftp.exe | grep --invert-match Windows

This shows you all the dependencies for the particular executable, in our case it shows the following.

ldd.exe lftp.exe | grep --invert-match Windows
        cygcrypto-1.0.0.dll => /usr/bin/cygcrypto-1.0.0.dll (0x3ffbf0000)
        cygwin1.dll => /usr/bin/cygwin1.dll (0x180040000)
        cygz.dll => /usr/bin/cygz.dll (0x3feb40000)
        cygexpat-1.dll => /usr/bin/cygexpat-1.dll (0x3ff980000)
        cygiconv-2.dll => /usr/bin/cygiconv-2.dll (0x3ff3c0000)
        cygintl-8.dll => /usr/bin/cygintl-8.dll (0x3ff360000)
        cygreadline7.dll => /usr/bin/cygreadline7.dll (0x3fede0000)
        cygncursesw-10.dll => /usr/bin/cygncursesw-10.dll (0x3ff120000)
        cygssl-1.0.0.dll => /usr/bin/cygssl-1.0.0.dll (0x3fecc0000)
        cyggcc_s-seh-1.dll => /usr/bin/cyggcc_s-seh-1.dll (0x3ff850000)

Long package list

autoconf, automake, bison, bzip2, cygwin-debuginfo, cygwin32, cygwin32-expat, cygwin32-libiconv, expat, expat-debuginfo, gcc-core, gcc-fortran, gcc-g++, gnutls, gnutls-devel, libexpat-devel, libexpat1, libiconv, libiconv-debuginfo, libiconv-devel, libiconv2, libmhash-devel, libmhash2, libreadline-devel, libreadline7, make, mhash-debuginfo, mingw-binutils, mingw-gcc-core, mingw-pthreads, mingw-runtime, mingw-w32api, mingw-zlib, mingw-zlib-devel, mingw-zlib1, mingw64-x86_64-binutils, mingw64-x86_64-gcc-core, mingw64-x86_64-gcc-fortran, mingw64-x86_64-gcc-g++, mingw64-x86_64-headers, mingw64-x86_64-runtime, mingw64-x86_64-winpthreads, mingw64-x86_64-zlib, openssl, openssl-devel, perl, perl-ExtUtils-PkgConfig, pkg-config, pkg-config-debuginfo, python, python-crypto, python-crypto-debuginfo, python-openssl, python-openssl-debuginfo, python-tkinter, zlib-devel, zlib0

Build and packaging script

#!/bin/bash
if [ $# -eq 0 ]
  then
    echo "&lt;script> name"
	exit
fi

if [ ! -f lftp.exe ];
then
 echo "lftp.exe doesn't exist run this file from where it exists"
 exit
fi
cwd=`pwd`
items=`ldd.exe lftp.exe | grep --invert-match Windows | cut -d " " -f3`
tmpdir=`mktemp.exe --directory`
echo "Creating temporary directory $tmpdir"
mkdir $tmpdir/bin
mkdir $tmpdir/etc
for item in $items
do
 echo "Copying LFTP dependency file: $item to $tmpdir/bin"
 cp $item $tmpdir/bin
done
items=`ldd.exe /bin/ssh.exe | grep --invert-match Windows | cut -d " " -f3`
for item in $items
do
 echo "Copying SSH dependency file: $item"
 cp $item $tmpdir/bin
done
items=`ldd.exe /bin/bash.exe | grep --invert-match Windows | cut -d " " -f3`
for item in $items
do
 echo "Copying BASH dependency file: $item"
 cp $item $tmpdir/bin
done
echo "Copying required files"
cp lftp.exe $tmpdir/bin
cp ../lftp.conf $tmpdir/etc
cp /bin/ssh.exe $tmpdir/bin
cp /bin/bash.exe $tmpdir/bin
cd $tmpdir
tar -cvzf ../$1.tar.gz .
mv ../$1.tar.gz $cwd
cd $cwd
echo "File generated in $cwd/$1.tar.gz"

OpenLayers draggable popups

Mon, 23 Sep 2013 09:45:00 +0000

How to implement a draggable popup in OpenLayers, to have the ability to decide which class will be draggable and which one will not be?

For a project I needed to be able to drag the OpenLayer popups, the problem was there were some implementations online, but I needed for the popup to be only draggable on the title, and for input elements not to interfere with the dragging process. Becase when I clicked on an input, textarea it suddenly stared dragging and it was a mess.

So the solution was to implement a control that will do all that for me.

The solution I came up was this, you can see the file on GitHub on OpenLayers.Control.DragPopup.js, the plugin requires Underscore and jQuery to work, when I have time I will rewrite them to be pure JavaScript only, but for now it works without any problems.

Let’s see some of the available options:

elementsIgnore

Which html elements should we ignore for dragging, this is useful so we don’t accidentally drag the popup on some of the elements inside the popup, like input elements.
selectorsIgnore

Which selectors do we ignore, this is an array that is used to check if we should move the element or not.
moveOnSelector

On which selector do we move, this can be a title or an icon, and we use this to determine if we should move or not.

To view a demo implementation, you can open this jsfiddle

Generating filename based on the android project name and version name when running ant

Sun, 16 Jun 2013 03:34:00 +0000

How to generate a filename based on the android project name and version when running ant, we will look at how the ant scripts should look like after the modifications

If you’ve ever wanted to generate the android project filename according to some data from your android manifest, you can use the ant build scripts to add the functionality needed.

If you don’t have your ant build script in your project directory you can run the following to command to create it along with some of the necessary files required.

android update project -p <android_project_path>

The original ant build script is like this:

<?xml version="1.0" encoding="UTF-8"?>
<project name="android_project_name" default="help">

    <property file="local.properties" />
    <property file="ant.properties" />

    <property environment="env" />
    <condition property="sdk.dir" value="${env.ANDROID_HOME}">
        <isset property="env.ANDROID_HOME" />
    </condition>

    <loadproperties srcFile="project.properties" />

    <fail
            message="sdk.dir is missing. Make sure to generate local.properties using 'android update project' or to inject it through the ANDROID_HOME environment variable."
            unless="sdk.dir"
    />

    <import file="custom_rules.xml" optional="true" />
    <import file="${sdk.dir}/tools/ant/build.xml" />

</project>

So we need to add the following to the ant build script, you can add all this above the line

<import file="${sdk.dir}/tools/ant/build.xml" />

This is the new configuration that you need to add to the ant script

    <xmlproperty
        collapseAttributes="true"
        file="AndroidManifest.xml"
        prefix="mymm" />

    <target name="custom-set-release-mode" >

        <property
            name="out.packaged.file"
            location="${out.absolute.dir}/${ant.project.name}-${mymm.manifest.android:versionName}-release-unsigned.apk" />

        <property
            name="out.final.file"
            location="${out.absolute.dir}/${ant.project.name}-${mymm.manifest.android:versionName}-release.apk" />
    </target>

    <target name="custom-set-debug-files" >

        <property
            name="out.packaged.file"
            location="${out.absolute.dir}/${ant.project.name}-${mymm.manifest.android:versionName}-debug-unaligned.apk" />

        <property
            name="out.final.file"
            location="${out.absolute.dir}/${ant.project.name}-${mymm.manifest.android:versionName}-debug.apk" />
    </target>

    <target
        name="debug"
        depends="custom-set-debug-files, android_rules.debug" />

    <target
        name="release"
        depends="custom-set-release-mode, android_rules.release" />

You can customize the files to provide any combination you want that you can access from either the AndroidManifest.xml or the properties files.

Using ant to generate javadoc for your android project

Sat, 15 Jun 2013 11:04:00 +0000

This process describes the generation of an android project using ANT, which will generate the project and also generate the javadoc documentation

If you want to generate JavaDoc for your project directly from the command line, you can easily do it by adding some extra configuration to the ant build.xml script.

If you don’t have your ant build script in your project directory you can run the following to command to create it along with some of the necessary files required.

android update project -p <android_project_path>

The original ant build script is like this:

<?xml version="1.0" encoding="UTF-8"?>
<project name="android_project_name" default="help">

    <property file="local.properties" />
    <property file="ant.properties" />

    <property environment="env" />
    <condition property="sdk.dir" value="${env.ANDROID_HOME}">
        <isset property="env.ANDROID_HOME" />
    </condition>

    <loadproperties srcFile="project.properties" />

    <fail
            message="sdk.dir is missing. Make sure to generate local.properties using 'android update project' or to inject it through the ANDROID_HOME environment variable."
            unless="sdk.dir"
    />

    <import file="custom_rules.xml" optional="true" />
    <import file="${sdk.dir}/tools/ant/build.xml" />

</project>

So we need to add the following to the ant build script, you can add all this above the line

<import file="${sdk.dir}/tools/ant/build.xml" />

This is the new configuration that you need to add to the ant script

<property name="docs.dir" location="javadoc" />
<property name="bin.dir" location="bin" />
<property name="source.dir" location="src" />
<property name="gen.dir" location="gen" />

<target
    name="javadoc"
    description="Generate JavaDoc documentation" >

    <xmlproperty
        collapseAttributes="true"
        file="AndroidManifest.xml"
        prefix="tm" />

    <mkdir dir="${docs.dir}" />

    <javadoc
        access="private"
        author="true"
        classpath="${sdk.dir}/platforms/${target}/android.jar"
        destdir="${docs.dir}"
        linkoffline="http://d.android.com/reference ${sdk.dir}/docs/reference"
        linksource="true"
        sourcepath="${source.dir};${gen.dir}"
        use="true"
        version="true" />

    <jar
        basedir="${docs.dir}"
        compress="${jar.compress}"
        jarfile="${bin.dir}/${tm.manifest.package}_${tm.manifest.android:versionName}_javadoc.jar" />
</target>

<target
   name="clean"
   depends="android_rules.clean" >

   <delete dir="${docs.dir}" />
</target>

Now we can run the following commands

ant clean

Will clean the directories for the android project and also the javadoc directory as it’s overridden in the target
ant javadoc

Will generate the JavaDoc in the ${docs.dir} It will also generate a jar file with the javadoc

Git alias, color and pretty printing

Fri, 14 Jun 2013 06:47:00 +0000

I’m tired of looking at the plain old log in git, so let’s see how we can spice it up a bit?

If you ever needed to have a prettier format of printing the git log, for easier changes, you can add this to the alias section of the .gitconfig file.

log-color = log --graph --full-history --all --color --pretty=format:'%x1b[31m%h%x09%x1b[32m %C(white)- %d%x1b[0m%x20%s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --date=relative

So after you add this line to the alias section executing git log-color on a git repo directory will give you the following output.

In our case for the node git repo: https://github.com/joyent/node

Websocket traffic forwarding using iptables

Tue, 30 Apr 2013 03:53:00 +0000

I’ve had a problem, I needed to route websocket trafic from internal server for external access, but I ran into a lot of problems, like not connecting, or the connections dropped, so let’s see how we can route the traffic successfully

The machines are all behind a iptables firewall.

I was using socket.io server in the background on multiple machines and I needed them all to be accessible from the outside through different ports.

The server is setup like this:

eth0      Link encap:Ethernet  HWaddr ##:##:##:##
          inet ##:##:##:##  Bcast:##:##:##:##  Mask:255.255.255.224
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:15151497 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11137070 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:7810856929 (7.8 GB)  TX bytes:4719777638 (4.7 GB)
          Interrupt:20 Memory:ec300000-ec320000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:50967 errors:0 dropped:0 overruns:0 frame:0
          TX packets:50967 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:9190223 (9.1 MB)  TX bytes:9190223 (9.1 MB)

vboxnet1  Link encap:Ethernet  HWaddr 0a:00:27:00:00:01
          inet addr:192.168.22.251  Bcast:192.168.22.255  Mask:255.255.255.0
          inet6 addr: fe80::800:27ff:fe00:1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:102350 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:20302629 (20.3 MB)

So to create the correct iptables rules, we do the following:

iptables -t nat -A PREROUTING -p tcp --dport <OUTSIDE_PORT> -j DNAT --to-destination <INTERNAL_IP_ADDRESS>:<INTERNAL_PORT>
iptables -t nat -A POSTROUTING -p tcp --dport <INTERNAL_PORT> -j MASQUERADE

So in my case I had:

iptables -t nat -A PREROUTING -p tcp --dport 31261 -j DNAT --to-destination 192.168.22.1:31261
iptables -t nat -A POSTROUTING -p tcp --dport 31261 -j MASQUERADE

Let explain what happens with these commands:

-j DNAT Destination-NAT (DNAT) - Changing the recipient

If you want to change the receipient of a packet, Destination NAT (DNAT) is your choice! DNAT can be used for servers running behind a firewall. Obviously the receipient has to be changed before any routing decisions are made, therefore DNAT is meaningful within the PREROUTING chain and the OUTPUT chain (for locally generated packets) only
-j MASQUERADE - Change sender to router’s IP-Adress

Using the MASQUERADE target every packet receives the IP of the router’s outgoing interface. The advantage over SNAT is that dynamically assigned IP addresses from the provider do not affect the rule, there is no need to adopt the rule. For ordinary SNAT you would have to change the rule every time the IP of the outgoing interface changes. As for SNAT, MASQUERADE is meaningful within the POSTROUTING-chain only. Unlike SNAT, MASQUERADE does not offer further options.
-p tcp The protocol type

Now the connection should be established

Hard links to a file

Sun, 21 Apr 2013 05:09:00 +0000

Describes the process of creating a hard link to file, and searching for hard links for any given file.

To create a hard link it’s easy just write

ln <target> <destination>

Because they are not symbolic links they won’t show up as links when you do

ls -l

So to find which files have links, you can do the following

find -type f -links +1

This will give you which files are hard linked, but will not tell you to which files they are hard linked too.

To find the which files are linked to which files you can use inodes, to find the inode of a file you simply can use the command stat.

stat <filename/directory>

This will give you an output similar to this one.

  File: `/tmp/tmp.file'
  Size: 122880    Blocks: 240       IO Block: 4096   file
Device: 815h/2069d	Inode: 1441793     Links: 20
Access: (1777/drwxrwxrwt)  Uid: (0/root)   Gid: (0/root)
Access: 2013-03-18 01:04:18.182327478 +0900
Modify: 2013-04-21 13:48:14.227089591 +0900
Change: 2013-04-21 13:48:14.227089591 +0900
 Birth: -

The inode number is there so now you can just search for that inode

ls -iR / | grep <inode_number>

Another way to find the files linked is to use this command but only if you know the file you want search for.

find / -samefile <full_path_to_filename>

SNS/SQS for node.js

Tue, 16 Apr 2013 03:24:00 +0000

How to interface SNS, SQS for node.js to AWS?

If you want to be able to create a SNS/SQS bindings from code and not to do it from the AWS Console with node.js you can do it like this.

var AWS = require('aws-sdk'), util = require('util');

// configure AWS
AWS.config.update({
    'region': 'us-east-1',
    'accessKeyId': 'akId',
    'secretAccessKey': 'secretKey'
});

var sns = new AWS.SNS().client;
var sqs = new AWS.SQS().client;

This will be enough to initialize the required data to be able to connect to the system.

First let’s create a topic:

sns.createTopic({
    'Name': 'demo'
}, function (err, result) {

    if (err !== null) {
        console.log(util.inspect(err));
        return;
    }

    console.log(util.inspect(result));

});

The following is the output:

{
    ResponseMetadata: {
        RequestId: 'bdceeca8-bbb6-511f-b186-a54e7c61fb0e'
    },
    TopicArn: 'arn:aws:sns:us-east-1:<AWSID>:demo'
}

Now that we have the TopicArn we can proceed to create the SQS Queue.

sqs.createQueue({
    'QueueName': 'demo'
}, function (err, result) {

    if (err !== null) {
        console.log(util.inspect(err));
        return;
    }

    console.log(util.inspect(result));

});

The output is:

{
    ResponseMetadata: {
        RequestId: '07d264eb-e8c5-5821-9720-2d0d215a8652'
    },
    QueueUrl: 'https://sqs.us-east-1.amazonaws.com/<AWSID>/demo'
}

Now we have the QueueUrl which we can use to get the ARN for the Queue

sqs.getQueueAttributes({
    QueueUrl: 'https://sqs.us-east-1.amazonaws.com/<AWSID>/demo',
    AttributeNames: ["QueueArn"]
}, function (err, result) {

    if (err !== null) {
        console.log(util.inspect(err));
        return;
    }

    console.log(util.inspect(result));

});

This will give us the QueueArn

{
    ResponseMetadata: {
        RequestId: '05be9746-9f9c-5c5a-94e6-dea0550f02f7'
    },
    Attribute: {
        Name: 'QueueArn',
        Value: 'arn:aws:sqs:us-east-1:<AWSID>:demo'
    }
}

Now we have everything we need to create a subscription and policy.

First let’s create the subscription

sns.subscribe({
    'TopicArn': 'arn:aws:sns:us-east-1:<AWSID>:demo',
    'Protocol': 'sqs',
    'Endpoint': 'arn:aws:sqs:us-east-1:<AWSID>:demo'
}, function (err, result) {

    if (err !== null) {
        console.log(util.inspect(err));
        return;
    }

    console.log(util.inspect(result));

});

The output from this query is:

{
    ResponseMetadata: {
        RequestId: '136057ff-0423-5f63-b42c-1e19c80948ad'
    },
    SubscriptionArn: 'arn:aws:sns:us-east-1:<AWSID>:demo:7825661f-130e-4e0c-bc29-9ad397660aaa'
}

This will enable us to publish to the SQS Queue, the only thing left now is to add the permissions that will enable the Topic to write to the SQS Queue.** **

var queueUrl = "https://sqs.us-east-1.amazonaws.com/<AWSID>/demo";
var topicArn = "arn:aws:sns:us-east-1:<AWSID>:demo";
var sqsArn = "arn:aws:sqs:us-east-1:<AWSID>:demo";

var attributes = {
    "Version": "2008-10-17",
    "Id": sqsArn + "/SQSDefaultPolicy",
    "Statement": [{
            "Sid": "Sid" + new Date().getTime(),
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "SQS:SendMessage",
            "Resource": sqsArn,
            "Condition": {
                "ArnEquals": {
                    "aws:SourceArn": topicArn
                }
            }
        }
    ]
};

sqs.setQueueAttributes({
    QueueUrl: queueUrl,
    Attributes: {
        'Policy': JSON.stringify(attributes)
    }
}, function (err, result) {

    if (err !== null) {
        console.log(util.inspect(err));
        return;
    }

    console.log(util.inspect(result));
});

And the response is:

{
    ResponseMetadata: {
        RequestId: '1487064f-38fa-55dd-a772-4fa9f5530f48'
    }
}

Now we have everything setup so when we publish something in a topic the queue will also be populated.

Maintaining multiple version of a software

Mon, 18 Mar 2013 13:15:00 +0000

Describes the process of maintaining multiple version of Eclipse on your linux machine

I’ve needed on a couple occasions to use a software with different version numbers.

On Ubuntu, the solution is pretty easy: update-alternatives

Let’s look at how we can keep multiple versions of eclipse on the system.

Download, and unpack eclipse in the opt folder, we have version 4.2.2

update-alternatives --install /usr/bin/eclipse eclipse /opt/eclipse/eclipse-SDK-4.2.2/eclipse 50

After this we can see that this command creates a symlink to the eclipse program, this way we can add multiple version of the program and switch really easy. This is applicable to all the application that run on Ubuntu.

Or you can even link the entire directory

update-alternatives --install /opt/eclipse eclipse /opt/.versions/eclipse/eclipse-SDK-4.2.2 500

Wireshark No Interfaces

Tue, 29 Jan 2013 09:28:00 +0000

How to fix the issue with Wireshark not showing any interface in linux?

If when you start Wireshark complains that you don’t have any interfaces just execute the following commands

sudo addgroup -quiet -system wireshark
sudo chown root:wireshark /usr/bin/dumpcap
sudo setcap cap_net_raw,cap_net_admin=eip /usr/bin/dumpcap
sudo usermod -a -G wireshark $USER

After this just start Wireshark and it you will have your interfaces back.

Automatic SVN incremental backup

Fri, 25 Jan 2013 04:03:00 +0000

How to create a simple incremental backup system, that you can restore at any time at a later time?

There are a lot of ways to create automatic incremental backups.

The simplest way is to use a post-commit hook that will automatically generate a backup every time we commit something to the repository.

And then we can always backup those files anywhere we want and we can recreate the SVN repository from those files.

Here is the script that will do that:

#!/bin/sh

PROJECT="project_name"
REPOS="$1"
REV="$2"
REVPAD=$(printf "%05d" "$REV")
FILE="${REVPAD}.svndump"

if [ ! -d "/backup/svn/$PROJECT/incremental/" ]; then
  mkdir -p /backup/svn/$PROJECT/incremental/
fi

svnadmin dump "$REPOS" --revision "$REV" --incremental > "/backup/svn/$PROJECT/incremental/${FILE}"

So every time we commit it’s gonna create a dump in the backup directory. And after that we can use any number of tools to sync that directory to any remote server we want to.

Buliding lftp from source with OpenSSL libraries

Fri, 25 Jan 2013 03:26:00 +0000

How to build the best ftp client for linux?

This is one of the best FTP client available, but it will take a little using to it as it’s command line and there is no GUI for it.

I’ve needed to connect to SSL sites but the lftp that comes with Ubuntu 10.04 doesn’t have SSL support compiled into it.

To check if your version has SSL support compiled into it:

ldd /usr/bin/lftp

linux-vdso.so.1 =>  (0x00007fffcc983000)
libgnutls.so.26 => /usr/lib/libgnutls.so.26 (0x00007f99eacc5000)
librt.so.1 => /lib/librt.so.1 (0x00007f99eaabd000)
libreadline.so.6 => /lib/libreadline.so.6 (0x00007f99ea87b000)
libutil.so.1 => /lib/libutil.so.1 (0x00007f99ea678000)
libncurses.so.5 => /lib/libncurses.so.5 (0x00007f99ea435000)
libdl.so.2 => /lib/libdl.so.2 (0x00007f99ea230000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f99ea019000)
libc.so.6 => /lib/libc.so.6 (0x00007f99e9c93000)
/lib64/ld-linux-x86-64.so.2 (0x00007f99eaf93000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f99e9a75000)
libtasn1.so.3 => /usr/lib/libtasn1.so.3 (0x00007f99e9864000)
libz.so.1 => /lib/libz.so.1 (0x00007f99e964d000)
libgcrypt.so.11 => /lib/libgcrypt.so.11 (0x00007f99e93d4000)
libgpg-error.so.0 => /lib/libgpg-error.so.0 (0x00007f99e91d0000)

This will give you a list of the dependencies libraries, see ldd for a more complete description.

So if there is no libssl library in the list that means you need to compile it from source.

apt-src install lftp

Now we need to modify the building rules. We need to edit the debian/rules file and add the following details to the configuration section.

--with-openssl=/usr/lib \
--with-socks=/usr/lib

We also need to install some libraries needed to compile.

apt-get build-dep lftp
apt-get install libsocks4 libsocksd0-dev libssl-dev

So the new configuration should look like this.

./configure \
--prefix=/usr \
--mandir=\$${prefix}/share/man \
--infodir=\$${prefix}/share/info \
--sysconfdir=/etc \
--with-pager=/etc/alternatives/pager \
--with-openssl=/usr/lib \
--with-socks=/usr/lib

So after this we need to compile the packages.

apt-src build lftp

And after a while you should have a package ready to compile. And we can install it with:

dpkg -i lftp*.deb

Getting the IP address of a running VirtualBox VM

Wed, 16 Jan 2013 14:15:00 +0000

To get the IP address of a running virtual machine we just need to do the following:

VBoxManage guestproperty get <UUID|Machine Name> "/VirtualBox/GuestInfo/Net/0/V4/IP"

This will give you the IP4 address of the Virtual Machine if it’s set.

Upgrade libssl on Ubuntu Lucid 10.04

Mon, 14 Jan 2013 10:25:00 +0000

If you ever needed to upgrade the libssl library on your system this is how you can do it.

First add the Precise source repository to your apt sources.

Now we just run:

apt-src install libssl1.0.0
apt-src build libssl1.0.0

Compiling will take a little time, but after compilation you will have the following files available:

libssl1.0.0_1.0.1-4ubuntu3_amd64.deb
libssl1.0.0-dbg_1.0.1-4ubuntu3_amd64.deb
libssl1.0.0-udeb_1.0.1-4ubuntu3_amd64.udeb
libssl-dev_1.0.1-4ubuntu3_amd64.deb
libssl-doc_1.0.1-4ubuntu3_all.deb
openssl_1.0.1-4ubuntu3_amd64.deb

So we can either install all the packages with dpkg or, we can, if you have read Creating your own local compiled packages repository you can just move those files to the folder described there and after that run the update script the new libssl will be available for installation in the system

Creating your own local compiled packages repository

Mon, 14 Jan 2013 09:50:00 +0000

I’ve had the problem, I needed to create a local repository that will contain all my packages pre compiled so I can just use the apt utility that comes with Debian based systems.

Execute the following commands to prepare your system for this.

mkdir -p /usr/local/deb```

Now let's create a new file called **update** in that directory
```bash
#!/bin/sh
echo "Updating the local repository in" `pwd`
find . -mindepth 1 -maxdepth 1 -type d \
   | cut -c3- \
   | xargs -n 1 -I DIRNAME dpkg-scanpackages DIRNAME /dev/null \
   | gzip -c9 >Packages.gz
[ $? -ne 0 ] \
   && echo "Update failed" 1>&2 \
   && [ `id -u` -ne 0 ] && echo "Maybe you've forgotten to sudo?" 1>&2

This file is to help us create a packaging list that should be used as a repository.

So in our example for Lucid we execute the following commands from /usr/local/deb directory:

mkdir -p lucid/packages

Now all the debian packages that we have we need to put in lucid/packages directory, and after that we need to run the update script from /usr/local/deb directory.

sudo ./update lucid

This will create the necessary structure and files so it can be recognized by apt .

Now we just need to add this repository to the apt sources list

echo /etc/apt/sources.list.d/local.list > deb file:/usr/local/deb/lucid/ ./

After this we run the update command and then we can just use apt to install or upgrade from the command line.

apt-get update
apt-get upgrade

Some resources of where you can implement this:

Compile qt 4.8.4 on Ubuntu 10.04 Lucid

Mon, 14 Jan 2013 07:18:00 +0000

Download the full installation from qt 4.8.4, and un-compress the file onto a separate directory.

Before running the installation run the following command to get all the dependencies

apt-get build-dep libqt4-core libqt4-dev

This will download and install the necessary libraries for qt4.

You can also review which libraries are required here.

If you want to build everything then just run

./configure

This will build everything but it will take a lot of time, if you are like me and you don’t need the examples, tutorials, demos run the following command

./configure \
       -debug-and-release \
       -opensource \
       -developer-build \
       -no-webkit \
       -nomake examples \
       -nomake demos \
       -nomake docs \
       -fast

After this one is finished and if there are no errors, you can proceed to running

make
sudo make install

Make sure you build the source code without root privileges and after that if you want to install it globally run the make install command with sudo

Intel Wireless N Cards

Thu, 03 Jan 2013 03:59:00 +0000

I’ve had a lot of problems with the Intel wireless cards, when connecting to Wireless N Routers.

The speed cuts off, or drops to 1MB/s, you can set the rate manually by issuing the following command, replace wlan0 with the equivalent for your system, but usually it’s wlan0 unless you have multiple wireless cards.

iwconfig wlan0 rate 54M

This is a temporary solution and it won’t solve the problem permanently.

To solve this permanently issue the following command, to create a new file:

echo "options iwlagn 11n_disable=1" > /etc/modprobe.d/intel_11n_disable.conf

After this we need issue the following commands:

update-initramfs -u

Then we need to either restart the system, or we can just issue the following commands to reinitialize the wifi driver.

modprobe -r iwlagn
modprobe iwlagn

Tracing jQuery trigger events

Wed, 02 Jan 2013 10:13:00 +0000

We will see how to modify the trigger event of jQuery so we can see everytime there is an event triggered, this is really usefull if you need debuging

I’ve had to see what trigger events have been fired when I was using jQuery, or to see if the trigger events I’ve done were working for a project.

So I had to modify the trigger event, but to also keep the functionality of jQuery the same so the following command gave me the stuff I needed to modify the trigger event.

Just write the following in developer tools of Chrome

$.fn.trigger
function ( type, data ) {
		return this.each(function() {
			jQuery.event.trigger( type, data, this );
		});
	}

So you modify this to the following, and now every-time there is a trigger event from jQuery you will be notified in the console.

$.fn.trigger = function ( type, data ) {
        return this.each(function() {
            console.log("trigger", type, data, this);
            jQuery.event.trigger( type, data, this );
        });
    };

Setting JAVA_HOME to proper value

Sun, 07 Oct 2012 21:51:00 +0000

There were a lot of times, when I needed to set up more than one java version. We should add the following line at the beginning of the file /etc/environment

JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")

This way every-time you start the system it will read which java has been selected by update-alternatives

Also if you change your java version often you can add it to your .bashrc file so everytime you login in a shell it will read and set JAVA_HOME to the proper directory.

Compile/Backport packages from source Ubuntu/Debian

Thu, 12 Apr 2012 09:52:00 +0000

I’ve run into several situations where the packages I want are not available on the distribution version I’m running.

One way to solve this is to upgrade to a newer distribution that contains the packages you want.

Or you can just compile them your self and install them after that. This is done in two steps, one is we need to fetch the sources and then we need to compile them.

As I’m on on Lucid 10.04 and I need a newer libnotify library, I will get it from Maveric sources repository and then compile it.

apt-src install libnotify
apt-src build libnotify

Or just run the following, it will have the same effect as the previous two lines:

apt-src install -b -i libnotify

After this you will have the following packages:

libnotify1_0.5.0-2ubuntu1_amd64.deb
libnotify-bin_0.5.0-2ubuntu1_amd64.deb
libnotify-dev_0.5.0-2ubuntu1_amd64.deb
libnotify-doc_0.5.0-2ubuntu1_all.deb

If you have read Creating your own local compiled packages repository you can just move those files to the folder described there and after that run the update script and then run:

apt-get upgrade

It will upgrade the version of libnotify to the one you just compiled.

Daily SVN Incremental Backup

Thu, 20 Oct 2011 08:14:00 +0000

Lately I’ve had a lot of repositories to deal with, so I wanted to back it up, beside doing the usual, creating a hotcopy I decided to create incremental backups of all the repositories, so we can restore them even if for some reason the hotcopy fails or the repository get’s corrupted, both of them the master and the hotcopy.

So I need to do this on windows, and on linux (ubuntu), but this should work on all the distributions of linux, provided you have the required programs installed, this is mostly for Cygwin, because usually the tools required come pre-installed on the distributions, and you also have to have installed Subversion and have it in your path.

#!/bin/bash

SCRIPT_NAME=`basename $0`

function help() {

    echo
    echo "Incremental dumping for SVN by Ilija Matoski (ilijamt@gmail.com)"
    echo "No command arguments detected, there need to be as following."
    echo "Usage: ${SCRIPT_NAME} <path_to_repo> <path_to_dump> <identifier>"
    echo
    echo "Parameters:"
    echo "  <path_to_repo> - Where is the repository located"
    echo "  <path_to_dump> - Where should we dump the repository"
    echo "  <identifier>   - Used to record the last recorded dump, always stored in the file with the"
    echo "                   dumps from the repo, so we can keep a track of where we at with dumping,"
    echo "                   also identifier is used to generate the dumps names."
    echo "  <compress>     - compress the output dump (default: false)"
    echo
    echo "Examples:"
    echo "  ${SCRIPT_NAME} test /backup/SVN/test test"
    echo "  ${SCRIPT_NAME} test /backup/SVN/test test true"
    echo

    exit -1

}

if [ ! $# -ge 3 ]; then
    help
fi

PATH_TO_REPO=$1
PATH_TO_DUMP=$2
IDENTIFIER=$3
COMPRESS=$4

# Check if they are empty and if they are quit
if [[ -z "$PATH_TO_REPO" || -z "$PATH_TO_DUMP" || -z "$IDENTIFIER" ]]; then
    help
fi

if [ "$COMPRESS" == "true" ]; then
    COMPRESS="true"
else
    COMPRESS="false"
fi

# Check if the repo path exists, and if it's a valid SVN repository
if [ ! -d "$PATH_TO_REPO" ]; then
    echo "Directory ${PATH_TO_REPO} doesn't exist."
    echo "Exiting."
    exit -1
fi

# Check if SVN folder is valid, check for format file in the directory
if [ ! -f "${PATH_TO_REPO}/format" ]; then
    echo "Probably not a valid SVN repository."
    echo "Exiting."
    exit -1
else
    echo "Probably a valid SVN repository detected. Continuing."
fi

# Check if destination exist and if we have writing permissions
# and if it doesn't then create it
if [ ! -d "$PATH_TO_DUMP" ]; then
    echo "Dump directory doesn't exist, creating a new directory: $PATH_TO_DUMP"
    mkdir -p "$PATH_TO_DUMP"
fi

# Check if the directory has writing permissions
if [ ! -w "$PATH_TO_DUMP" ]; then
    echo "No writing permissions."
    echo "Exiting."
    exit -1
fi

# Get the full paths if we have relative paths in the command line
PATH_TO_REPO=`cd $PATH_TO_REPO; pwd`
PATH_TO_DUMP=`cd $PATH_TO_DUMP; pwd`
IDENTIFIER_PATH="$PATH_TO_DUMP/$IDENTIFIER"
DUMP_FILE_MASK="$IDENTIFIER_PATH"
# Check if the identifier exists
if [ ! -f "$IDENTIFIER_PATH" ]; then
    # Doesn't exist, so we create it and set it to revision 0
    echo "-1" > "$IDENTIFIER_PATH"
fi

CYGWIN=`uname -o`

# Check if we are running cygwin, for windows
if [ "$CYGWIN" == "Cygwin" ]; then
    CYGWIN="true"
else
    CYGWIN="false"
fi

# Set up some stuff for cygwin
if [ "$CYGWIN" == "true" ]; then
    repoDrive="${PATH_TO_REPO:10:1}"
    REPO_URL=`echo $PATH_TO_REPO | sed "s/\/cygdrive\/${repoDrive}/${repoDrive}:/g"`
    REPO_DIR="$REPO_URL"
    REPO_URL="file:///${REPO_URL}"
else
    REPO_URL="file:///$PATH_TO_REPO"
    REPO_DIR="$PATH_TO_REPO"
fi

# Get the last revision from the directory repository
REPO_LAST_REVISION=`svn info $REPO_URL | grep "^Revision: " | sed "s/Revision: //g"`
REPO_DUMPED_REVISION=`cat "$PATH_TO_DUMP/$IDENTIFIER"`

# check if the call succeded
if [[ $REPO_LAST_REVISION  != [0-9]* ]]; then
    echo "Cannot fetch information from SVN repository"
    exit -1
fi

# Increment it by one number from the last revision
# so we don't re dump the last one again
REPO_DUMPED_REVISION=$(($REPO_DUMPED_REVISION+1))

echo "Settings"
echo "  Compress: ${COMPRESS}"
echo "  Directory: ${REPO_URL}"
echo "  Dump: ${PATH_TO_DUMP}"
echo "  Identifier: ${IDENTIFIER}"
echo "  Cygwin: ${CYGWIN}"
echo
echo "Repository"
echo "  Revisions: ${REPO_LAST_REVISION}"
echo "  Dumping from: ${REPO_DUMPED_REVISION}"
echo

if [ $REPO_DUMPED_REVISION -gt $REPO_LAST_REVISION ]; then
    echo "ERROR: Dumped revision bigger than the revision in the repository"
    echo "Exiting."
    exit -1
fi

echo "Starting to dump"

list=$(seq "$REPO_DUMPED_REVISION" 1 "$REPO_LAST_REVISION" )

for rev in $list
do

    padding_rev=`printf "%08d" $rev`
    dump_file="$DUMP_FILE_MASK.rev.$padding_rev.svndump"

    if [ "$COMPRESS" == "true" ]; then
        dump_file="${dump_file}.gz"
    fi

    echo "  Dumping revision $rev of $REPO_LAST_REVISION"

    if [ "$COMPRESS" == "true" ]; then
        svnadmin dump "$REPO_DIR" --revision "$rev:$rev" --incremental --deltas -q | gzip > $dump_file
    else
        svnadmin dump "$REPO_DIR" --revision "$rev:$rev" --incremental --deltas -q > $dump_file
    fi

    # put the new dumped revision in the file
    echo $rev > "$IDENTIFIER_PATH"

done

echo "Done."

It’s really easy to use this to create a incremental backup of your SVN repository. If you need help just write, without any arguments:

./incremental_backup.sh

And it will show you the help/usage screen.

Domain and Web hosting on different servers

Thu, 08 Sep 2011 09:08:00 +0000

If you want a website you need two things, the domain name and the web hosting. I ran into a small problem a while ago, the usual method of integrating them together is simple, just point the domain nameservers to the web hosting nameservers and it’s finished, after the DNS fully propagates you can assign it to your account.

My problem was this I had a web hosting account on Hostmonster and a I had a “.dk” domain registered on BB-Online, and it turns out I can’t point the nameservers to the hostmonster ones, because for some reason they can’t communicate between them.

And, I was wondering what to do, but the solution turned out to be pretty simple, you just edit your DNS Zone File for the domain and point it to the server that you use in Hostmonster, it’s in cPanel on the side where it says IP Address, in my case it says “Shared IP Address”.

After that you just sit back and wait for the DNS to update, between 24-48 hours.

This method should be applicable almost on any web hosting.

Move the steam apps installation folder

Thu, 28 Jul 2011 20:17:00 +0000

Well like a lot of people, I had a problem that I have no space on my install partition, at best around 2GB-3GB, so I wanted to move the SteamApp folder to a different partition which had a lot more space, but I couldn’t find anywhere in setting where to set install directory or where to change the install directory.

You have only two choices if you have installed Windows Vista or Windows 7, there is an utility called:

mklink

This creates a link between folders on different partitions, so you can have installed some stuff on your main partition and some on a different partition, so what I did is:

Shutdown Steam if it’s ruining
Create a new folder on the partition where you have space, in my case I have on partition D so I decided to create a folder there called SteamApps
Copy everything from the folder called **steamapps **in the installation directory of steam, to the new directory called **SteamApps, **on your new partition, in my case partition D
Delete the directory called steamapps in the installation directory as you wont need it anymore
Go to the steam installation directory and execute the command to create a junction between folders mklink /J steamapps D:\SteamApps

And that’s it, but what about all the other people who don’t use Windows Vista or Windows 7, well there is a solution for them too, and it can be used on Windows XP too, but the prerequisite is that the partitions need to be NTFS, or it wont work.

There is an application that was released with Windows 2000, that can do the same thing mklink can

linkd

And don’t ask me where to find it, you can easily find it with a search on your favorite search provider.

Unix timestamp in Intersystems Caché

Fri, 22 Jul 2011 08:07:00 +0000

I wanted to have Unix Timestamp in Intersystems Caché, but their timestamp is a little different than the one you are used to in other programming languages, their timestamp is “ddddd,sssss.fff”, while Unix Timestamp is defined as the number of seconds elapsed since midnight Coordinated Universal Time (UTC) of January 1, 1970, not counting leap seconds.

So to convert Intersystems version to Unix Timestamp we use this:

({CACHETIMESTAMP} - 47117 * 86400 + $PIECE({CACHETIMESTAMP},",",2))

So to convert Unix Timestamp to Intersystems version we use this:

(({UNIXTIMESTAMP}\86400 + 47117)_","_({UNIXTIMESTAMP} # 86400))

We can combine these and create our own macros that will do the conversion on the fly when compiling.

#define toUnixTimestamp(%ct) (%ct - 47117 * 86400 + $PIECE(%ct,",",2))
#define toCacheTimestamp(%ut) ((%ut\86400 + 47117)_","_(%ut # 86400))
#define addToUnixTimestamp(%ct, %a) ($$$toUnixTimestamp(%ct) + %a)

Remove old kernels from debian based systems

Wed, 20 Jul 2011 22:29:00 +0000

I’ve always hated that there is no simple way to remove all the kernels you don’t use from the Debian/Ubuntu system.

But fortunately it’s really easy to do so, from the command line.

First thing reboot your system and start the kernel you want to keep, in this instance I want to keep the newest only, because I’m dual-booting between Windows and Ubuntu, I just keep only the latest kernel.

So to do that we just write in the console, you can also use purge instead of remove.

dpkg -l linux-* | awk '/^ii/{ print $2}' | grep -v -e `uname -r | cut -f1,2 -d"-"` | grep -e [0-9] | xargs sudo apt-get -y remove

Or:

dpkg -l linux-* | awk '/^ii/{ print $2}' | grep -v -e `uname -r | cut -f1,2 -d"-"` | grep -e [0-9] | xargs sudo apt-get -y purge

So maybe I should explain what is happening here, though if you have used linux you know most if not all of these commands.

awk - used for textual data manipulation.

grep - filters out the data from the input

cut - remove sections from each line of files

uname - shows system information

dpkg - package manager in debian distributions

xargs - build and execute command lines from standard input

So basically I use dpkg to get the linux kernel files, and we pipe it to awk to filter out everything but the package name, and we use uname and cut to get the kernel version only, and when we put it together we select all linux kernel files except the one provided by uname .

And that is it, the best way to learn it is to try and experiment by yourself.

Show empty categories in WordPress

Sat, 16 Jul 2011 10:22:00 +0000

I wanted to show all my categories on the screen on the sidebar widget, because if the category doesn’t have any posts inside the categories doesn’t show.

So, one way of fixing it is to assign an dummy post to all categories, and they will show then, but instead of doing that we can add a filter that will always show our categories, without assigning a post to all the categories in WordPress.

Open the file called “/wp-includes/default-filters.php” in the WordPress root directory, and add this snipped to the end of the file.

function force_widget_cat_args($cat_args) {
    $cat_args['hide_empty'] = 0;
    return $cat_args;
}

add_filter( 'widget_categories_args', 'force_widget_cat_args' );

Hello

Tue, 12 Jul 2011 20:13:00 +0000

Hi, I’ve finally decided to set up my blog.

So here it is and hope you guys have fun here.

I’m thinking of starting this blog off, with some kind of android app, like show you how to build it from scratch, or something else, but we will see, the night is still young.