Obtaining and running oprofile

Февраль 20th, 2014 | Posted by elenst in Pensieve | Testing - (Комментарии отключены)

It’s not that easy to get oprofile nowadays… Shame, really.

Obtaining

For Precise, found it here: https://launchpad.net/ubuntu/+source/oprofile/0.9.6-1.3ubuntu1
Plain deb files, e.g. for amd64

https://launchpad.net/ubuntu/+source/oprofile/0.9.6-1.3ubuntu1/+build/2614177/+files/libopagent1_0.9.6-1.3ubuntu1_amd64.deb

https://launchpad.net/ubuntu/+source/oprofile/0.9.6-1.3ubuntu1/+build/2614177/+files/oprofile_0.9.6-1.3ubuntu1_amd64.deb

Amazingly,
dpkg --install libopagent1_0.9.6-1.3ubuntu1_amd64.deb oprofile_0.9.6-1.3ubuntu1_amd64.deb
worked like a charm.

For Wheezy, used the Squeeze repo:

deb http://ftp.ru.debian.org/debian squeeze main

Also had to install keys

sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com ...

and then apt-get install just worked.

Running

Installing it in the Wheezy VM was useless. It turns out that oprofile does not work in a VM.
Well, technically it works, if you set sudo modprobe oprofile timer=1, but then it returns something like that:


1043 75.8545 /data/repo/10.0/sql/mysqld my_timer_cycles
313 22.7636 /data/repo/10.0/storage/federatedx/ha_federatedx.so federatedx_io_mysql::mark_position(st_federatedx_result*, void*)
6 0.4364 /data/repo/10.0/sql/mysqld code_state
3 0.2182 /no-vmlinux /no-vmlinux
2 0.1455 /data/repo/10.0/sql/mysqld _db_enter_
...

No much use.

On Precise, it required running

sudo bash -c "echo 0 > /proc/sys/kernel/nmi_watchdog"

and then it actually worked.

To run (for mysqld):


sudo opcontrol --init
sudo opcontrol --setup --separate=lib,kernel,thread --no-vmlinux
sudo opcontrol --start-daemon
sudo opcontrol --start

Then do stuff in mysqld that needs to be profiled. Then

sudo opcontrol --dump
sudo opreport --demangle=smart --symbols --long-filenames --merge \
tgid path-to-mysqld | head -n 20
sudo opcontrol --stop
sudo opcontrol --deinit
sudo opcontrol --reset

Creating a local deb repository with MariaDB packages

Май 22nd, 2013 | Posted by elenst in Pensieve | Testing - (Комментарии отключены)

Based on Daniel’s email and my own experience.

Variables to set:

DEBIAN=/path/to/mkrepo-debian.sh # /path/to/mariadb-tools/buildbot/mkrepo-debian.sh)
UBUNTU="/path/to/mkrepo-ubuntu.sh" # (/path/to/mariadb-tools/buildbot/mkrepo-ubuntu.sh)
version="mariadb-10.0.2" # change to whatever makes sense
tree="10.0" # change to whatever tree you are using
tarbuildnum="3544" # change to the correct tar build number
archive_dir="/media/backup/archive/pack" # this is on hasky, but it should be local, so I need to create some $(archive_dir)/$(tree) folder locally, and copy build-$(tarbuildnum) from hasky into it.

Modify mkrepo-debian.sh and mkrepo-ubuntu.sh scripts to use my GPG key.
If needed, modify the scripts to exclude distributions that aren’t needed, especially if they are not present on hasky.

Might need to install reprepro if it’s not there yet:
apt-get install reprepro

mkdir -v /path/to/where/i/want/${version}/repo
cd /path/to/where/i/want/${version}/repo
eval $(gpg-agent --daemon)
${DEBIAN} debian ${archive_dir}/${tree}/build-${tarbuildnum}
${UBUNTU} ubuntu ${archive_dir}/${tree}/build-${tarbuildnum}

I actually didn’t run eval $(gpg-agent —daemon) , don’t even have it installed, and it still worked all right, or so it seemed.

Then, add

deb file:///path/to/where/i/want/${version}/repo/ubuntu precise main
deb-src file:///path/to/where/i/want/${version}/repo/ubuntu precise main

(and comment the previous one if there was any).
Replace ubuntu and precise with whatever you’re on.

Run

sudo apt-get update

It might complain about GPG key, then make sure it’s installed as described in http://elenst.ru/pensieve/it-bits/creating-and-installing-a-local-gpg-key-for-testing-purposes/

Then install as usual.

Creating and installing a local GPG key for testing purposes

Май 22nd, 2013 | Posted by elenst in Pensieve | Testing - (Комментарии отключены)

No fancy GUI is necessary. The base gpg is installed on Ubuntu by default.

gpg --gen-key

Use default type, size and duration.
Enter the name and the email address (I suppose it doesn’t matter which as long as it’s for testing only).
Type O to create the key.
Enter a passphrase twice.
Move your mouse like crazy, but try to spare the effort, it might take long and you’ll get tired. When it’s enough, it will tell you it’s enough. Until it does, keep moving. Don’t stop for long, or you’ll have to repeat the exercise.

The file pubring.gpg will appear in $HOME/.gnupg/

Finally,

sudo apt-key add $HOME/.gnupg/pubring.gpg

Now I can run apt-get update with repos where something is signed with this key.

Packages to get MariaDB and tests up and running

Январь 18th, 2013 | Posted by elenst in MariaDB | Pensieve | Testing - (Комментарии отключены)

yum

It’s often pain to guess package names when you need to install stuff on, lets say, CentOS. So there is a list, although maybe not full, of what I needed to get another VM build and run MariaDB server and to execute at least some tests on it (all done via yum install):

cmake
gcc
ncurses-devel
bison
g++
gcc-c++
aclocal
automake
libtool
perl-DBD-MySQL
gdb
libaio-devel
openssl-devel

Same in one line, for lazy me:
sudo yum install cmake gcc ncurses-devel bison g++ gcc-c++ aclocal automake libtool perl-DBD-MySQL gdb libaio-devel openssl-devel

To install bzr (if it’s not in the official repo):

su -c ‘rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-2.noarch.rpm’
(check the architecture)

and then can use yum install
bzr

Another story (taken from http://wiki.bazaar.canonical.com/Download):
su -c 'rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm'

In newer distributions bzr seems to be already in place.

To install jemalloc:

Search for recent versions of
jemalloc-3.6.0-1.el6.x86_64
jemalloc-devel-static-3.6.0-23.1.x86_64
jemalloc-devel-3.6.0-1.el6.x86_64

download them, install.

To install pcre-devel:

Search for recent versions of
pcre
pcre-devel

download them, install.

apt-get

Pretty much the same stuff, but few names are different:

bzr
cmake
gcc
make
ncurses-dev
bison
g++
automake
libtool
gdb
valgrind
libaio-dev
libssl-dev

libdbi-perl
libdbd-mysql-perl

libjemalloc-dev
libjemalloc1

libpcre3
libpcre3-dev

The same in one line:
sudo apt-get install bzr cmake gcc make ncurses-dev bison g++ automake libtool gdb valgrind libaio-dev libssl-dev libdbi-perl libdbd-mysql-perl libjemalloc1 libjemalloc-dev libpcre3 libpcre3-dev

To build PAM plugin:

libpam0g-dev

zypper

bzr
cmake
gcc
make
ncurses-devel
bison
gcc-c++
automake
libtool
gdb
valgrind
libaio-devel
perl-DBD-mysql

The same in one line:
sudo zypper install cmake gcc make ncurses-devel bison gcc-c++ automake libtool gdb valgrind libaio-devel perl-DBD-mysql

Installing Java on Ubuntu — meant to be easy, right?

Август 22nd, 2012 | Posted by elenst in Pensieve - (Комментарии отключены)

Actually, I was installing Cassandra. But this is another story, I’m afraid will be coming soon.

So, Cassandra wants Java, specifically
Depends: openjdk-6-jre-headless (>= 6b11) but it is not going to be installed

JDK wants tzdata-java, no specifics:
Depends: tzdata-java but it is not going to be installed

tzdata-java wants tzdata, but…
Depends: tzdata (= 2011k-1) but 2012b-0ubuntu0.11.10 is to be installed

Fun, isn’t it? It’s not like I’m trying to pull Java from some obscure repository.

Some digging revealed that 2011K-1 lives on Ubuntu Main repository, while 2012b (or even 2012e) on Ubuntu Proposed Main repository. There wasn’t the Proposed Main repository on my source list… I wonder where it was pulling it from. (Maybe it was from the Cassandra repo? I didn’t think about it back then.) Anyway, even although my source list is nearly pristine, I prefer to think that it was a problem with my configuration, rather than suspect that the Ubuntu main repository offers conflicting packages.

And ‘anyway’ again, since we’re already playing with 2012e, I’d rather they give me both of that.
So, I added the Proposed Main repository. No luck, still the same incompatibility. I tried to remove the Main repository. No luck again. Well, there are many default Ubuntu repositories in the sources.list, I didn’t feel like playing with them all, especially since apt-get update is annoyingly slow. So, I replaced /etc/apt/sources.list with a dummy one which contained only the Proposed Main repository:
deb http://archive.ubuntu.com/ubuntu/ oneiric-proposed main

And voila!

Stupid workaround, but fast if you know what you want, but don’t know exactly what you’re doing.

I even remembered to restore the initial sources.list afterwards, but of course I forgot to ‘apt-get update’ it, and spent a couple of fancy minutes trying to figure out why now Cassandra refuses to install, pretending it had never been there…

GPG error: BADSIG .. Ubuntu Archive Automatic Signing Key

Август 21st, 2012 | Posted by elenst in Pensieve - (Комментарии отключены)


GPG error: http://ru.archive.ubuntu.com oneiric Release: The following signatures were invalid:
BADSIG <..> Ubuntu Archive Automatic Signing Key
A error occurred during the signature verification. The repository is not updated and the previous index files will be used.
GPG error: http://extras.ubuntu.com oneiric Release: The following signatures were invalid:
BADSIG <..> Ubuntu Extras Archive Automatic Signing Key
Failed to fetch http://extras.ubuntu.com/ubuntu/dists/oneiric/Release
Some index files failed to download. They have been ignored, or old ones used instead.

Solution (well, workaround):
sudo rm /var/lib/apt/lists/* && sudo rm /var/lib/apt/lists/partial/* && sudo apt-get update

Discussion:
https://bugs.launchpad.net/ubuntu/+source/synaptic/+bug/863306

I took it from here

Deceiving xfs: if you want the real deal, go for xfsprogs

Август 17th, 2012 | Posted by elenst in Pensieve - (Комментарии отключены)

You think you are smart when you run sudo mkfs -j -t ext3 /dev/sdb1? But you’re not. Look what happens:
mke2fs 1.41.11 (14-Mar-2010)
Filesystem label=
OS type: Linux
...

It’s still same old mke2fs, also known as “make me extN”. Quick look at man mkfs says that what I really want is mkfs.xfs. Only, I don’t have what I want.

How does one get what they want on Ubuntu? Ask apt-get. The question is what to ask. It’s not like I can say sudo apt-get install mkfs.xfs, can I? One can dream… How about sudo apt-get install xfs, it looks like an obvious choice? And it does install!

Spoiler: don’t trust obvious choices.

Still no mkfs.xfs. So, what did we get?
xfs(1): X font server
facepalm.

So, google it!

However opensource and free it is, every time I type in a search engine ‘Download <something>’, I feel like I’m cheating. anyway, the first link goes here

And now, sudo apt-get install xfsprogs… Finally.

sudo mkfs.xfs /dev/sdb1

mkfs.xfs: /dev/sdb1 appears to contain an existing filesystem (ext3).
mkfs.xfs: Use the -f option to force overwrite.

Yeah, tell me about it — I’ve already formatted to ext3, remember?

sudo mkfs.xfs -f /dev/sdb1

blah-blah-blah
sudo mount /dev/sdb1 /mnt
df -kT
/dev/sdb1 xfs 1039392 4256 1035136 1% /mnt

Profit!

Need to remove the wrong XFS. X font server, my goodness.

aio-max-nr in general and “InnoDB: Error: io_setup() failed with EAGAIN” in particular

Август 3rd, 2012 | Posted by self in MariaDB - (Комментарии отключены)

The problem many MySQL/MariaDB 5.5+ users are painfully aware of:

InnoDB: Using Linux native AIO
InnoDB: Warning: io_setup() failed with EAGAIN. Will make 5 attempts before giving up.
InnoDB: Warning: io_setup() attempt 1 failed.
InnoDB: Warning: io_setup() attempt 2 failed.
InnoDB: Warning: io_setup() attempt 3 failed.
InnoDB: Warning: io_setup() attempt 4 failed.
InnoDB: Warning: io_setup() attempt 5 failed.
InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts.
InnoDB: You can disable Linux Native AIO by setting innodb_native_aio = off in my.cnf
InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: Completed initialization of buffer pool
mysqld got signal 11 ;

There is no news that disabling InnoDB native AIO is not exactly the best possible option. It’s also not a secret that the alternative is increasing aio-max-nr if possible. What is always news for me is how it’s done…
So,

to see the current value:
cat /proc/sys/fs/aio-max-nr
65536

to set the new one:
sudo sysctl fs.aio-max-nr=262144

And yeah, it has already been fixed in MariaDB: replication tests crash… But still.

Btw, while we are here — what is aio-max-nr, anyway?

aio-nr is the running total of the number of events specified on the io_setup system call for all currently active aio contexts. If aio-nr reaches aio-max-nr then io_setup will fail with EAGAIN. Note that raising aio-max-nr does not result in the pre-allocation or re-sizing of any kernel data structures.

Source: http://www.kernel.org/doc/Documentation/sysctl/fs.txt

grub2: what to edit and what not

Июль 22nd, 2012 | Posted by self in Pensieve - (Комментарии отключены)

With all these changes after the 1st grub, and considering how seldom I have to do it, every time it’s pain to remember. So:

/boot/grub/grub.cfg — watch, don’t touch.
/etc/default/grub — primary defaults file
GRUB_DEFAULT=0 — default menu entry by the position in the list
GRUB_DEFAULT="foo" — default menu entry by the exact name (at least they say so, didn’t try myself)
GRUB_TIMEOUT=10 — timeout

sudo update-grub2 — to rebuild the config file from various defaults files.

Add/remove a passphrase for an SSH key

Июль 19th, 2012 | Posted by self in Pensieve - (Комментарии отключены)

Got tired of searching for it every time, on some reason can never remember that it’s done by keygen…
ssh-keygen -p [-P old_passphrase] [-N new_passphrase] [-f keyfile]