Solarwinds Observability tips&tricks

Solarwinds is known for their tool Appoptics. Sadly it has been deprecated and considered legacy.

They have released a new tool called Observability which performs the same tasks, but with improvements and changes.
Although the price has not improved, infact they hiked it up by double .

Either way, it’s still a good alternative to other APMs (application performance monitor) such as New Relic.

One thing I noticed is their documentation is lacking.

Their install instructions for Linux and PHP for example looks like this:

curl -sSO https://agent-binaries.cloud.solarwinds.com/apm/php/latest/solarwinds-apm-php.sh

sh solarwinds-apm-php.sh –service-key=yourservicekey:tag –collector=apm.collector.eu-01.cloud.solarwinds.com

systemctl restart <php.service>
systemctl restart <web_server.service>

The installed doesn’t mention it, and they have hid the so files behind non public directory service service key.
By default it installs the so-library for the current active php-cli is installed, which means you only get one PHP version library installed.

To install Observability for other PHP versions you simply change your consoles default PHP version, and it’ll pull the right version when you run the solarwinds-apm-php.sh script.

In Debian it’s update-alternatives –config php
select the version you want and re-run the script.

Dont forget to restart your PHP services (fpm) and webserver!

VMWare slutar tillhandahålla fria utvärderingslicenser

VMWare som sålts till Broadcom slutar nu tillhandahålla fria licenser för utvärdering och labb. Läs mer på https://kb.vmware.com/s/article/96168?lang=en_US

Adminor kan hjälpa till att migrera last från VMware till andra moln, t.ex. Proxmox eller Hyper-V. Kontakta oss så hjälper vi gärna till.

Vill man själv migrera virtuella VMware maskiner till PRoxmox kan man göra detta genom att följa stegen på denna blogg. https://nicolas.busseneau.fr/en/blog/2021/07/esxi-to-proxmox-migration

Azure and Azure files MMAP broken

We found out (or rather re-discovered) that Azure Files CIFS implementation does not correctly support mmap function that is used in some softwares.

It has been previously reported on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900821 but we thought it might be worth clarifying this for people spending a lot of time googling why their apache2 fails to deliver files larger than 1MB.

The default setting in apache2 does not mmap files smaller than 1MB hence why the issue is not noticed on small files or files not hosted on the CIFS volume. Large files from the affected volume will however cause a ”Read error at byte X/ X (Bad address). Retrying.” error if you try to wget or download the files via apache2 on a web app using Azure Files mounted CIFS storage (path mapping).

Some searches might lead you to investigate apache2 file limitations preventing files larger than 1MB from beint transmitted.
LimitRequestBody is not a fix for this issue.

Scenario:
Azure web app or VM with Apache2 or other mmap using software fails to read files properly from Azure Files mounted ”path mapping” volumes which uses CIFS.

Workaround:
EnableMMAP off

This configuration line can be done per affected Directory, global apache2 config or in the vhost.

Important security notification: Linux remote crash vulnerability

We’ve been notified that there is a new remote crash vulnerability many Linux systems .

The CVE has yet to be publicly released, it has just been reserved so far: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11477
The register has published more details: https://www.theregister.co.uk/2019/06/17/linux_tcp_sack_kernel_crash/
Other cloud hosting vendors have published steps on how to mitigate this flaw and so is Adminor AB.

A patch to linux kernel will be issued by different vendors, meanwhile a mitigation of these attacks is to disable tcp_sack (tcp selective acknowledgement) .
It’s possible that the recent reboot of systems already have mitigation for this exploit but we have not been notified of such by upstream vendor as the exploit is still not entirely released.

We recommend that you disable tcp_sack as a pre-caution.
TCP sack is used to speed up TCP transfer by allowing computers to tell the server how much data is left to be sent. This should have minimal impact on normal operations but we still recommend monitoring for any negative performance impacts

Command to run which should not require a system reboot:
echo 0 > /proc/sys/net/ipv4/tcp_sack


To make the change persistent across reboots a command such as the following can be run:echo ’net.ipv4.tcp_sack = 0’ >> /etc/sysctl.conf

We recommend enabling tcp_sack when a kernel patch has been issued and system rebooted.
Please let us know if you need assistance .

The GDPR letter from hell

This GDPR letter is a worst case scenario of someone making a GDPR info request under the new regulations.

We recommend studying it, and discussing within your organization to practice and be ready for a worst-case letter like this.

Some examples where GDPR can be used as a ”weapon” against companies is in social media outreach campaigns, a lot of users would be encouraged to send such request in order to tie up an organizations resources.

Let us know if we can be of help to prepare you for this.

Dear Sir/Madam:

I am writing to you in your capacity as data protection officer for your company. I am a customer of yours, and in light of recent events, I am making this request for access to personal data pursuant to Article 15 of the General Data Protection Regulation. I am concerned that your company’s information practices may be putting my personal information at undue risk of exposure or in fact has breached its obligation to safeguard my personal information pursuant to <latest nasty cybersecurity event or thing in the news>.

I am including a copy of documentation necessary to verify my identity. If you require further information, please contact me at my address above.

I would like you to be aware at the outset, that I anticipate reply to my request within one month as required under Article 12, failing which I will be forwarding my inquiry with a letter of complaint to the <appropriate data protection authority>.

Please advise as to the following:

1.   Please confirm to me whether or not my personal data is being processed. If it is, please provide me with the categories of personal data you have about me in your files and databases.

a.   In particular, please tell me what you know about me in your information systems, whether or not contained in databases, and including e-mail, documents on your networks, or voice or other media that you may store.

b.   Additionally, please advise me in which countries my personal data is stored, or accessible from. In case you make use of cloud services to store or process my data, please include the countries in which the servers are located where my data are or were (in the past 12 months) stored.

c.   Please provide me with a copy of, or access to, my personal data that you have or are processing.

2.   Please provide me with a detailed accounting of the specific uses that you have made, are making, or will be making of my personal data.

3.   Please provide a list of all third parties with whom you have (or may have) shared my personal data.

a.   If you cannot identify with certainty the specific third parties to whom you have disclosed my personal data, please provide a list of third parties to whom you may have disclosed my personal data.

b.   Please also identify which jurisdictions that you have identified in 1(b) above that these third parties with whom you have or may have shared my personal data, from which these third parties have stored or can access my personal data. Please also provide insight in the legal grounds for transferring my personal data to these jurisdictions. Where you have done so, or are doing so, on the basis of appropriate safeguards, please provide a copy.

c.   Additionally, I would like to know what safeguards have been put in place in relation to these third parties that you have identified in relation to the transfer of my personal data.

4.   Please advise how long you store my personal data, and if retention is based upon the category of personal data, please identify how long each category is retained.

5.   If you are additionally collecting personal data about me from any source other than me, please provide me with all information about their source, as referred to in Article 14 of the GDPR.

6.   If you are making automated decisions about me, including profiling, whether or not on the basis of Article 22 of the GDPR, please provide me with information concerning the basis for the logic in making such automated decisions, and the significance and consequences of such processing.

7.   I would like to know whether or not my personal data has been disclosed inadvertently by your company in the past, or as a result of a security or privacy breach.

a.   If so, please advise as to the following details of each and any such breach:

                   i.    a general description of what occurred;

                   ii.    the date and time of the breach (or the best possible estimate);

                   iii.    the date and time the breach was discovered;

                   iv.    the source of the breach (either your own organization, or a third party to whom you have transferred my personal data);

                    v.    details of my personal data that was disclosed;

                    vi.    your company’s assessment of the risk of harm to myself, as a result of the breach;

                    vii.    a description of the measures taken or that will be taken to prevent further unauthorized access to my personal data;

                    viii.    contact information so that I can obtain more information and assistance in relation to such a breach, and

                     ix.    information and advice on what I can do to protect myself against any harms, including identity theft and fraud.

b.   If you are not able to state with any certainty whether such an exposure has taken place, through the use of appropriate technologies, please advise what mitigating steps you have taken, such as

                      i.    Encryption of my personal data;

                      ii.    Data minimization strategies; or,

                      iii.    Anonymization or pseudonymization;

                      iv.    Any other means

8.   I would like to know your information policies and standards that you follow in relation to the safeguarding of my personal data, such as whether you adhere to ISO27001 for information security, and more particularly, your practices in relation to the following:

a.   Please inform me whether you have backed up my personal data to tape, disk or other media, and where it is stored and how it is secured, including what steps you have taken to protect my personal data from loss or theft, and whether this includes encryption.

b.   Please also advise whether you have in place any technology which allows you with reasonable certainty to know whether or not my personal data has been disclosed, including but not limited to the following:

                     i.    Intrusion detection systems;

                     ii.    Firewall technologies;

                     iii.    Access and identity management technologies;

                     iv.    Database audit and/or security tools; or,

                     v.    Behavioural analysis tools, log analysis tools, or audit tools;

9.   In regards to employees and contractors, please advise as to the following:

a.   What technologies or business procedures do you have to ensure that individuals within your organization will be monitored to ensure that they do not deliberately or inadvertently disclose personal data outside your company, through e-mail, web-mail or instant messaging, or otherwise.

b.   Have you had had any circumstances in which employees or contractors have been dismissed, and/or been charged under criminal laws for accessing my personal data inappropriately, or if you are unable to determine this, of any customers, in the past twelve months.

c.   Please advise as to what training and awareness measures you have taken in order to ensure that employees and contractors are accessing and processing my personal data in conformity with the General Data Protection Regulation.

Yours Sincerely,

I. Rate

Source: https://www.linkedin.com/pulse/nightmare-letter-subject-access-request-under-gdpr-karbaliotis

Ubiquiti trådlöst spridningsnät i glesbygd

I September var jag uppe i Norrland och installerade bynät.
Valet föll på Ubiquiti airmax länkar p.g.a. svåra markförhållanden (ägarfrågor, avstånd & kostnadsbild).
Kostnaden för mast-installationen är ungefär samma som vad ett hushålls fiberdragning hade kostat om man anlitar en professionell fiberinstallatör.
Byn förses ändå med radiolänk från kommunen så fiberdragningen hade ändå bara varit lokal.
 
Med 5G nätverk runt hörnen så är det en fråga hur mycket krut man ska lägga ned, något man får ta ställning till då när Telia räknar med att glesbyggden ska få tillgång till det (2025) och då kanske airmax också ska uppgraderas. 5G nätverk är tänkt att ge vanliga konsumenter upp till 100Mbit/s bredband via telefonen fast man säger samtidigt att fiber och alternativa anslutningar är bra alternativ. Så jag antar att de själva ser 5G som ett komplement.
Tills dess finns det ju ändå inget som stoppar individuella hushåll att gräva ned fiber till byns internetknytpunkt om möjligheten finns.
Kapaciteten i nätet är byggt så att hushåll ska kunna garanteras minst 100Mbit/s i båda riktningar vid normala radioförhållanden även när andra nyttjar bandbredd, men om behovet finns kan man få upp till 330Mbit/s upp eller ned, eller 165Mbit/s i båda riktningar.
4 sektor antenner sprider 90 grader i var sin riktning (dvs 360).
Resultatet hittills är 6 hushåll uppkopplade via airmax trådlösa länkar. 2st företag samt att minst 3 hushåll till ska kopplas upp. Byns internetuppfart mot kommunen skrämdes dessutom upp i dubbla hastigheten.
Våra egna mätningar visade att den gamla länken presterar som mest 25Mbit/s (totalt) medan den kommunlänken vi satte upp i mast klarar upp till 100Mbit/s. Det är något som man kan öka till 450Mbit/s om kommunen använder den nya generationens utrustning.
I  takt med att fler får upp ögonen för bredbandet så vill fler ansluta och då kommer behovet att finnas. Ska bli spännande att se hur det utvecklar sig.
Väder och vind är ett problem som kan ställa till det. Hård kyla och massor med snö/is på vinter, åska och regn på sommar. Heta soliga dagar med mera.
Men Ubiquiti har god renommé  samt har byggt sina grejer för att klara detta.
Kanske ökar behovet av bredband så att man sätter press på kommunen att uppgradera radiolänkarna som förser byn med uppfarten. De är trots allt 2 generationer äldre än det vi installerade för byns hushåll.

DKIM och SPF record

Adminor erbjuder SPF record.
För vår server Smallfoot gäller följande SPF record:

”v=spf1 mx ip4:46.253.202.12/32 ip4:46.253.202.13/32 ip4:46.253.205.58/32 -all”

För DKIM kontakta oss så kan vi slå igång det.

DKIM och SPF används för att identifiera att avsändarserver har rätt att skicka mail. På så vis minskar man risken för att klassa mail felaktigt som SPAM. Korrekt uppsatta SPF/DKM record är därför viktigt för att undvika problem med utskick av mail.

Let’s encrypt for varnish?

Do you have a site that’s accelerated with varnish but noticed that there is no native SSL support for Varnish?

No problems!
You can use a bunch of different methods to terminate SSL .
In this post I’m not going to be posting a bunch of configuration or setup steps. But discuss the caveats of terminating SSL .

Let’s say you are using drupal or wordpress.
Your current setup probably looks something like this:

varnish –> apache OR nginx backend -> application (wordpress/drupal) .

With Let’s encrypt you’re going to want to setup an SSL terminator. In the past I’ve recommended using ”pound” as an SSL terminator, but due to the slow development cycles I’ve moved towards nginx or haproxy.

Of the two I’d setup nginx SSL terminator in most cases as the Let’s encrypt certbot supports nginx natively for issuing and renewing SSL certificates.

Haproxy is awesome if you plan to use multiple backends or caches. If you use haproxy you probably know what you are doing but the problem with Let’s encrypt is that you have to run certbot in standalone mode with the ”certonly” variable on the commandline .

Haproxy will be configured to pass the acme-challenge to the standalone daemon that certbot launches. The renewal process will also use the forwarded requests .

With nginx it’s as simple as just creating the cert using certbot and configuring SSL proxy onwards to varnish.

We assume that you have previously managed to configure apache mod_rpaf , mod_remoteip or nginx to handle x-forwarded-for to provide the right IP to the web application.

One big issue that we’ve seen on customer installations when adding let’s encrypt support to already running setups is the fact that the backend application has no clue about https protocol. This sometimes causes forwarding loops or problems with loading http:// resources over a https:// connection (modern webrowsers raises alerts and refuses to load resources outside the https:// connection if it’s properly configured).

It’s important that the x-forwarded-for and protocol is passed on in all steps of the chain.

With haproxy you use option forwardfor and x-forwarded-proto.
Nginx needs to do the same.

On the backend side you can force SSL in a number of different ways.
The easiest way in php is to force https. Drupal and WordPress have a number of plugins to do so, you can also edit the php config files to force SSL.

For example in wp-config.php :
define(’FORCE_SSL_ADMIN’, true);
// in some setups HTTP_X_FORWARDED_PROTO might contain
// a comma-separated list e.g. http,https
// so check for https existence
if (strpos($_SERVER[’HTTP_X_FORWARDED_PROTO’], ’https’) !== false)
$_SERVER[’HTTPS’]=’on’;

Do you need help to add SSL and let’s encrypt for your current varnish setups? Feel free to contact us 

Adminor also offers ready-made VM images with Let’s encrypt varnish proxies for our VPS customers.

Supersnabb WordPress installation på Adminor SSDVPS

Det första du behöver är en SSDVPS från Adminor (även om det går bra med andra leverantörer också så rekommenderar vi våra tjänster)

SSDVPS:en ska köra Ubuntu 14.04 LTS för att denna guide ska vara kompatibelt.
Vi kommer använda oss utav Zach Adams guide för att installera en miljö med följande komponenter:

Percona DB (MySQL) (Looking for MariaDB? Try this)
HHVM (Default PHP Parser)
PHP-FPM (Backup PHP Parser)
Nginx
Varnish (Running by default)
Memcached and APC
Clean WordPress Install (Latest Version)
WP-CLI

 

Installation

  1. SSH:a till din nyligen skapade server, lägg till nödväntiga apt paket om dessa redan inte är installerade:
    sudo apt-get install software-properties-common
  2. Lägg till Ansible med sudo add-apt-repository ppa:ansible/ansible
  3. Uppdatera Apt med sudo apt-get update && sudo apt-get upgrade
  4. Installera Git oc Ansible med sudo apt-get install ansible git
  5. Klona detta repository med git clone https://github.com/zach-adams/hgv-deploy-full/
  6. Flytta in till  cd hgv-deploy-full
  7. Editera hosts filen och ändra yourhostname.com till ditt eget hostname. Om du har fler siter på denna server så lägg till varje domän på en ny rad.
  8. Editera yourhostname.com filen in host_vars katalogen till ditt eget hostname. Om du vill installera fler siter på denna server så ska du kopiera den nuvarande och byta namn på den till den andra sitens domän .
  9. Ändra site specifik information inklusive lösenord innuti hostname filen i host_vars katalogen
  10. Kör Ansible med sudo ansible-playbook -i hosts playbook.yml -c local. Får du några fel så kontakta oss så kanske vi kan hjälpa till.
  11. Ta bort den klonade git katalogen från din server med rm -rf hgv-deploy-full/
  12. Kör /usr/bin/mysql_secure_installation för att installera MySQL och säkra det. Ditt root-lösenord är tomt från början.
  13. Starta om Varnish och Nginx med: sudo service varnish restart && sudo service nginx restart
  14. Du bör nu vara klar! En ny WordPress installation som kör HHVM och Varnish bör nu vara klar i hostname/s!

 

Hur man installerar en Ny Site / Hostname

Dessa steg funkar bara om du installerat med metoden ovan. Ta alltid en backup på din server innan du gör ändringar!

  1. Ta en backup
  2. Följ steg 1-6 ovan
  3. När du kommer till din hosts fil följ samma steg MEN dinkludera inga tidigare installationer av WordPress eller hostnames, bara de nya du vill installera.
  4. Upprepa detta för din host_var katalog
  5. Följ steg  9-12 och får du några problem så kan du kontakta oss så kanske vi kan hjälpa till.

Hur man stänger av Varnish (använd bara Nginx)

Om du får problem att ändra eller får problem med backend när du använder Varnish så kan du stänga av det och bara använda Nginx. Du bör fortfarande få relativt god prestanda. Så här gör du det:

  1. Öppna varje konfiguration i Nginx för siter du har installerat på din server med kommandot: sudo nano /etc/nginx/sites-available/your-hostname.com
  2. Ändra listen = 8080; till listen = 80;
  3. Gör detta för alla siter som är installerade på servern
  4. Stoppa Varnish och Starta om Nginx med sudo service varnish stop && sudo service nginx restart
  5. Du bör nu vara klar! Har du ingen cache plugin för wordpress installerat så rekommenderar vi att du skaffar en.

Växla från HHVM tillbaka till PHP-FPM

Din Nginx konfiguration bör växla automatiskt till PHP-FPM om det uppstår problem med HHVM, du kan däremot växla manuellt om du behöver göra det:

  1. Öppna din Nginx konfigurationsfil med  vim|emacs|nano /etc/nginx/sites-available/( Your Hostname )
  2. Ändra följande sektion i slutet av filen:
    location ~ \.php$ {
        proxy_intercept_errors on;
        error_page 500 501 502 503 = @fallback;
        fastcgi_buffers 8 256k;
        fastcgi_buffer_size 128k;
        fastcgi_intercept_errors on;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_pass hhvm;
    }
  1. Ändra fastcgi_pass hhvm; till fastcgi_pass php;
  2. Starta om Nginx med sudo service nginx restart
  3. Du bör nu köra PHP-FPM! Kontrollera med phpinfo(); i en php fil

Vill du att vi installerar åt dig så erbjuder vi konfigurationstjänst. Kontakta oss på [email protected] för offert!