/sbin/ip route add … RTNETLINK answers: File exists

Posted on by programmingnotepad

When starting opevpn I encountered strange issue. VPN was connecting but I could not access services behind it.
What I found in its log was:

Thu Oct 16 12:15:50 2014 /sbin/ip route add 10.1.1.0/24 via 10.1.1.1
RTNETLINK answers: File exists

It turned out the newly added route was conflicting with other route (a leftover from messing up vpn-things).

First, I removed the conflicting rule:

sudo route del -net 10.1.1.0 netmask 255.255.255.0

Then I manually added the route the vpn was trying to add (but I think it’s optional and simply restarting the VPN would do the thing)

/sbin/ip route add 10.1.1.0/24 via 10.1.1.1

Starting on system startup and respawning apps with upstart in ubuntu

Posted on by programmingnotepad

Configuring

Following is an example config file for starting app on system startup and auto-respawning.
File name should be put in /etc/init, its name should end with .conf

The name of the task is derived from file name. For file name taskName.conf, the task is called taskName.

start on runlevel [2345]

#!upstart
description "your app description"

# Start a job as username, a member of usergroup
setuid username
setgid usergroup

# set required environments (what you normally do with export VARIABLE=VALUE)
env ENV_TO_SET = environment\ value

# run the app
exec /home/myuser/bin/myapp

# Restart the process if it dies with a signal
# or exit code not given by the 'normal exit' stanza.
respawn

# Give up if restart occurs 10 times in 90 seconds.
respawn limit 10 90

Running

To start a task run:

$ sudo start taskName

And to stop it:

$ sudo stop taskName

More info: http://upstart.ubuntu.com/cookbook/

Notes

When you get some strange errors reading files, you might need to add an env which defines locale, e.g.:

env LC_ALL=pl_PL.UTF-8

For running java apps, remember to specify JAVA_HOME

setting up iptables to forward 80 (http) and 443 (https) to other ports

Posted on by programmingnotepad

If you want to serve http content from application run by a non-root user, you can set up iptables to forward http and https ports to “user space ports”.

Setting up

Assuming your app uses port 8080 for http and 8081 for https, configure iptables as follows:

$ sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080
$ sudo iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 8081

Saving and restoring

Test the configuration and if it works save it, e.g. with:

sudo iptables-save > /etc/iptables/rules

Now you have to add reading iptables configuration on system startup.
If your network is configured manually you can add iptables-restore < /etc/iptables/rules to your /etc/network/interfaces:

auto eth0
iface eth0 inet dhcp
        pre-up iptables-restore < /etc/iptables/rules

If your /etc/network/interfaces is generated automatically, you can create a new file in /etc/network/if-pre-up.d/ with following content:

#!/bin/bash

iptables-restore < /etc/iptables/rules
exit 0

importing eclipse android (ADT) project to Android Studio

Posted on by programmingnotepad
  1. In eclipse: generate gradle build files (via export -> android). Details: http://developer.android.com/sdk/installing/migrate.html
  2. In console:

    • change gradle version in gradle/wrapper/gradle-wrapper.properties to 1.9
    • change plugin version to 0.7 in build.gradle

    Details: http://stackoverflow.com/questions/19485981/unable-to-import-eclipse-project-to-android-studio

  3. Import into Android Studio

Mongo DB cheat sheet

Posted on by programmingnotepad

Quickstart

Start the server (by default on localhost:27017):

mongod [--dbpath /path/to/db/directory]

Start the client:

mongo

(you can write JavaScript in it! Reverse search works, it’s quite usable.)

Insert value:

> db.mycollection.insert({"name": "John", "lastname": "Smith", "age": 21})

 

Finding documents

Get single document from collection

> db.mycollection.findOne(filter, projection)

Condition and filter are optional.
Without the condition a random document from the collection will be returned. Without the filter all the fields will be returned.
_id is always included.
E.g.

> db.mycollection.findOne({"age":21}, {"name": true, "_id" : false})
{ "name" : "John" }

Get multiple documents:

> db.mycollection.find(filter, projection)

Without filter returns all documents.

Filters:

  1. $gt, $gte, $lt, $lte
  2. $exists
  3. $type – from bson spec, e.g. string -> 2 ($type: 2) (mongodb doc)
  4. $regex – perl-style regular expressions (perl 5 doc)
  5. $or – accepts an array of alternatives.
  6. $and – accepts an array of conditions. Might be usable if you want to put more than one condition for one field
  7. $all – accepts an array of values. Matches arrays which contain all of the provided values
  8. $in – accepts an array of values. Matches if attribute equals/contains one of the provided values
  9. $near – for a field which contains two values and has a 2d index, find the document with values closest to given values (result is ordered by distance to target values)
  10. . – searching for attributes of sub-documents. Works also for arrays. E.g. {“user.contact.email”: “me@example.com” }

$near can be also used for spherical search (location needs to be [longitude, lattitude]):

db.runCommand({geoNear: 'stores', near:[50, 50], spherical:true, maxDistance:numberOfRadians}

Cursors

creating a cursor

var c = db.mycollection.find()

coursor has hasNext() and next() methods

operations on coursors (in the order they are applied)

  • sort(sort_definition) – sort results, e.g. sort({name: -1})
  • skip(2) – skip number of results
  • limit(13) – limit the number of returned docs to 13

To stop liquibase from printing out cursor contents after its definiton or after modifications on it, you can write:

var c = db.mycollection.find(); null;

Sort and limit are not processed in the order they are defined. They are merged and then applied (on the server side).

Notes

  • if attribute’s value is an array, finding with {attribute: “value”} checks whether the array contains the value (not if the attribute is equal to value)
  • “normal” find for attributes which contain objects finds only exact matches (including attribute order in the sub-document)

Usage:

db.scores.find({score: {$gt: 20, $lte: 40}, middleName: {$exists:false}})
db.scores.find( { $or: [{"score": {$gt: 20, $lte: 25}}, {"middleName": {middleName: "John"}}, {"middleName": "Andrew"} ] } )

Aggregate

db.mycollection.aggregate([aggregate_def1, aggregate_def2, ...])
aggregate_defs are applied from left to right, the result of first is piped to second and so on
Where aggregate_def is e.g.:

{
  $group: {
    _id: "$key_attribute_name",
    result_attr_name:{$sum:1}
  }
}

For example: db.products.aggregate([{$group:{"_id":"$category", "num_products":{"$sum":1}}}])

Aggregation methods:

  • $project – select reshape, maps 1-1
  • $match – filters documents
  • $group – aggregation – grouping documents
  • $sort
  • $skip
  • $limit
  • $unwind – expand array attribute

Aggregation functions:

  • $sum
  • $avg
  • $min
  • $max
  • $push – create array of results
  • $addToSet – create array of distinct results
  • $first/$last

To have a compound aggregation key use: {_id: {"label1" : "$keyAttr1", "label2" : "$keyAttr2"}}

Update

To overwrite whole document (only _id will be kept untouched, the rest of the doc is removed and new attributes are added):

db.scores.update(filter, new_document)

To selectively update an attribute

  • $set – sets value of an attribute (update or insert). Works also with arrays, then the key should be: "attribute.index", e.g. "comments.13" updates/inserts 14th element into comments
  • $inc – increase the value of the given attribute by given number. If there’s no such attribute, one will be created with value = incrementation step
  • $push – add element at the end of an array
  • $pushAll – adds mulptiple elements at the end of the array
  • $addToSet – adds element iff it it’s not yet in the array
  • $pop – remove the last element of an array (given the value 1, with -1 it removes first one).
  • $pull(All) – remove specified value(s) from array
  • $unset – removes attribute from document.
    Usage: db.catalog.update({newField: "aanba"}, {$unset:{price : ""}}) price value in the $unset is ignored.

Update parameters(passed in the third parameter to update):

  • upsert – when set to true will create a new doc if no doc matches the filter. It will not only add the values from update, but also add the values from filter if it can determine them
  • multi – modify all documents matching filter (by default only first found is modified)

Warning: While single document update or removal is atomic, updating (and removing) multiple documents is not.
E.g.

db.scores.update(filter, {$set: {score: 30}})
db.scores.update(filter, {$inc: {score: 5}})
db.scores.update(filter, {$pop: {someArray: -1}}

Remove

db.mycollection.remove(filter) – remove matching document
db.mycollection.remove() – remove all documents
db.mycollection.drop() – much faster

Indexes

Index for _id is created by default.

If given collection has a multi-key index on attr1 and attr2, values for each of the attributes can be an array, but they cannot be an array at the same time.

I.e. you can insert {attr1: "value", attr2: ["some", "array"]} and {attr1: ["some", "array"], attr2: "value"} but cannot insert:
{attr1: ["some", "array"], attr2: ["yet", "another", "array"]}

Creating
db.coll.ensureIndex({attribute: order[, attr2: order2, ...]} [,{unique:true [, dropDups:true]}])

where order is +/-1

Unique index
db.coll.ensureIndex(attributes, {unique:true [, dropDups:true]}])

By default null == null for unique index, so only one row w/o value is allowed for an indexed attribute.

To make it work SQL-like use (works also for non-unique indexes):

db.coll.ensureIndex(attributes, {unique:true, sparse:true})

If you sort documents by attribute on which you have a sparse index, documents that have no value for the given attribute will be omitted!
The same with find – you cannot search for documents w/o attribute with such index

To create an index in background, add background:true to parameters. It doesn’t block writing, but works much slower (upto 5 times slower) than foreground.

Listing

All indexes for the databse:

db.system.indexes.find()

More details for indexes on a specific collection:

db.collection.getIndexes()
Removal
db.collection.dropIndex({attribute: order})
Hinting

In mongo shell: append .hint({a:1, b:1}) to query.
If you don’t want mongo to use index: .hint({$natural:1})
In java, alternatively to use BasicDBObject defining hint index, one can use a string, e.g. “a_1_b_-1_c_1”

Other

To count the elements instead of reading them use:

> db.scores.count(filter)

Check the name of the current database:

> db
test

To get the info how the query was performed (usage of indices, etc), add .explain() at the of the query.

To modify document and get its version before or after modification use findAndModify

To read the status of latest query run:

db.runCommand({getLastError: 1})

To check how the query is run use .explain()
Notes
_id cannot be changed (the primary key, generated if not provided, unique within collection)

To log queries that take longer than 2 ms: mongod --profile 1 --slowms 2

db.system.profile.find({nsi:/test.foo}).sort({ts:1}) // searching in test db in foo collection ?
db.system.profile.find({millis: {$gt:1}}).sort({ts:1})
db.getProfilingStatus()
db.setProfilingLevel(1,4) // enabled, slowms: 4
db.setProfilingLevel(0) // disabled
mongotop – slow queries by collection

mongostat -> idx miss – how many index lookups are made in memory
javadoc, etc for mongo: api.mongodb.org

speed up (e.g.) mvn jetty:run by disabling annotation scanning on jetty startup

Posted on by programmingnotepad

When your app gets large and you don’t use annotation based configuration, to improve its’ startup time, you can do the following:

1. add metadata-complete=”true” to your web.xml header:

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="3.0" xmlns="http://java.sun.com/xml/ns/javaee"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
   metadata-complete="true">

2. add absolute-ordering inside the file.

   <absolute-ordering/>
</web-app>

Which .gitignore rule is making my file ignored?

Posted on by programmingnotepad

When you add a file to git and you get:
The following paths are ignored by one of your .gitignore files: your-file

you can check which .gitignore rule makes the path ignored using:
git check-ignore -v your-file

git check-ignore checks if file (or list of files) is ignored by git, -v makes the output more verbose, adding a .gitignore file name and line.