Community

Hi guys, I'm writing this post because I'm having problems with my nimbula infra. I installed director 1.5 and when I try to list the contents of an image list, for example, I get the following error:

nimbula-api -d -a http://10.32.0.5 -u /root/root -p nimbula list imagelist "/root/root/Lista1" -f json
POST http://10.32.0.5/authenticate/
Request headers:
Content-Length: 74
Host: 10.32.0.5
Content-Type: application/nimbula-v2+json
Accept-Encoding: gzip;q=1.0, identity; q=0.5
Accept: application/nimbula-v2+json
Data:
{"result": null, "password": "nimbula", "uri": null, "user": "/root/root"}

204 No Content
Response headers:
set-cookie: nimbula={"identity": "{\"realm\": \"test\", \"value\": \"{\\\"customer\\\": \\\"root\\\", \\\"realm\\\": \\\"test\\\", \\\"entity_type\\\": \\\"user\\\", \\\"expires\\\": 1331581887.029047, \\\"user\\\": \\\"/root/root\\\", \\\"groups\\\": [\\\"/root\\\"]}\", \"signature\": \"APN5ySxyNXQEw/OasA58DeHPT/kiwG8lKRyQp1tTYE4YWUEKKoQdEmrmznV7o620nwwUoWK3koKxrLT6u55XbcY7a2WFPlZgMrlqadIUFnh2PI2AZUrWA5fuPEC5xuAgAFZ5EviFk31ZGhc0aKlb3fL/wTqkBNJQaQZ1NavDXBzf2bCX/JCAHf9cs1Jl60DJLyyZNuuo1kwdSBLTuczziUcHGgqfeFQaK6AVHM6N4rfn19ZvZYyi4cwOxPoBD7APfXy3vwBVvyleJnI2Ki773KnYLXHYFddjv3LbBQBy7zk71mdTIZpNpLeVpuY8F5tjUIJnNG329KrES/FUYxt9gV8MHrFx\"}"}; Expires=Mon, 12-Mar-2012 17:51:27 GMT; Path=/
expires: Mon, 12 Mar 2012 16:51:27 GMT
vary: Accept
server: nginx/1.1.1
connection: keep-alive
date: Mon, 12 Mar 2012 16:51:27 GMT
content-type: application/nimbula-v2+json
Data:
<unknown length of data>

GET http://10.32.0.5/imagelist/root/root/Lista1/
Request headers:
Host: 10.32.0.5
Cookie: nimbula={"identity": "{\"realm\": \"test\", \"value\": \"{\\\"customer\\\": \\\"root\\\", \\\"realm\\\": \\\"test\\\", \\\"entity_type\\\": \\\"user\\\", \\\"expires\\\": 1331581887.029047, \\\"user\\\": \\\"/root/root\\\", \\\"groups\\\": [\\\"/root\\\"]}\", \"signature\": \"APN5ySxyNXQEw/OasA58DeHPT/kiwG8lKRyQp1tTYE4YWUEKKoQdEmrmznV7o620nwwUoWK3koKxrLT6u55XbcY7a2WFPlZgMrlqadIUFnh2PI2AZUrWA5fuPEC5xuAgAFZ5EviFk31ZGhc0aKlb3fL/wTqkBNJQaQZ1NavDXBzf2bCX/JCAHf9cs1Jl60DJLyyZNuuo1kwdSBLTuczziUcHGgqfeFQaK6AVHM6N4rfn19ZvZYyi4cwOxPoBD7APfXy3vwBVvyleJnI2Ki773KnYLXHYFddjv3LbBQBy7zk71mdTIZpNpLeVpuY8F5tjUIJnNG329KrES/FUYxt9gV8MHrFx\"}"}
Accept-Encoding: gzip;q=1.0, identity; q=0.5
Accept: application/nimbula-v2+json

500 Internal Server Error
Response headers:
date: Mon, 12 Mar 2012 16:51:27 GMT
connection: keep-alive
content-type: application/json
content-length: 71
server: nginx/1.1.1
Data:
{"message": "An internal error occurred.", "reference": "AXJF6UI6SHNL"}

APIUncaughtExceptionError: An internal error occurred.
Reference: AXJF6UI6SHNL


So I executed nimbula-test-services on one of the nodes and here is the output I received:

bash-4.1# nimbula-test-services
2012-03-12 16:40:54,021 INFO: Discovering nodes...
Please enter /root/root password for node lookup:
2012-03-12 16:40:58,435 INFO: Checking for the API...
2012-03-12 16:40:58,439 INFO: Authenticating...
2012-03-12 16:40:58,659 INFO: Discovered 3 nodes in 10.32.0.0/26
2012-03-12 16:40:58,659 INFO: Discovered 3 nodes
2012-03-12 16:40:58,659 INFO: Checking for SSH on all nodes...
2012-03-12 16:40:58,661 INFO: Checking for bIC on all nodes...
2012-03-12 16:40:58,664 INFO: Checking for bIC master in DNS...
2012-03-12 16:40:58,683 INFO: Checking DNS...
2012-03-12 16:40:58,684 INFO: Checking PXE stack...
2012-03-12 16:40:58,686 INFO: Checking Gluster servers...
2012-03-12 16:40:58,693 INFO: Checking Hadoop NameNode...
2012-03-12 16:40:58,695 INFO: Checking Hadoop Secondary NameNode...
2012-03-12 16:40:58,697 INFO: Checking Hadoop DataNodes...
2012-03-12 16:40:58,698 INFO: Checking ZooKeepers...
2012-03-12 16:40:58,705 INFO: Checking DRBD...
2012-03-12 16:40:58,708 INFO: Checking Postgres...
2012-03-12 16:40:58,710 INFO: Checking memcache server...
2012-03-12 16:40:58,712 INFO: Checking bAuth...
2012-03-12 16:40:58,715 INFO: Checking bAuthFront...
2012-03-12 16:40:58,718 INFO: Checking bUser...
2012-03-12 16:40:58,720 INFO: Checking bUserFront...
2012-03-12 16:40:58,723 INFO: Checking bImage...
2012-03-12 16:40:58,725 INFO: Checking bFederated...
2012-03-12 16:40:58,728 INFO: Checking bNet...
2012-03-12 16:40:58,730 INFO: Checking bSecSite...
2012-03-12 16:40:58,732 INFO: Checking bServiceSite...
2012-03-12 16:40:58,734 INFO: Checking bStorageSite...
2012-03-12 16:40:58,737 INFO: Checking cluster controllers...
2012-03-12 16:40:58,752 INFO: Checking sanity of cluster controller 10.32.0.10
2012-03-12 16:40:58,752 INFO: Checking site controller...
2012-03-12 16:40:58,760 INFO: Checking for bNode on all nodes...
2012-03-12 16:40:58,782 INFO: Checking sanity of node 10.32.0.12
2012-03-12 16:40:58,805 INFO: Checking sanity of node 10.32.0.11
2012-03-12 16:40:58,827 INFO: Checking sanity of node 10.32.0.10
2012-03-12 16:40:58,827 INFO: Checking for bSecNode on all nodes...
2012-03-12 16:40:58,830 INFO: Checking for flow-engine on all nodes...
2012-03-12 16:40:58,833 INFO: Checking DHCP services...
2012-03-12 16:40:58,833 ERROR: According to bIC master, 3 instances of bdhcp are running (should be 2)
2012-03-12 16:40:58,833 INFO: Checking vService managers...
2012-03-12 16:40:58,833 INFO: Checking storage nodes...
2012-03-12 16:40:58,833 ERROR: According to bIC master, 3 instances of bstoragenode are running (should be 2)
2012-03-12 16:40:58,833 INFO: Checking site monitoring master...
2012-03-12 16:40:58,833 ERROR: According to bIC master, 0 instances of gmetad-site are running (should be 1)
2012-03-12 16:40:58,837 ERROR: No response from DNS servers 10.32.0.3, 10.32.0.4 for monitor.test.nimbula
2012-03-12 16:40:58,838 INFO: Checking cluster monitoring masters...
2012-03-12 16:40:58,846 INFO: Checking for gmond on all nodes...
2012-03-12 16:40:58,851 INFO: Checking NTP servers...
2012-03-12 16:40:58,856 INFO: Checking for site logging servers...
2012-03-12 16:40:58,860 INFO: Checking for cluster logging servers...
2012-03-12 16:40:58,866 INFO: Checking for local logging servers...
2012-03-12 16:40:58,867 INFO: Checking frontend webserver...
2012-03-12 16:40:58,871 INFO: Authenticating...
2012-03-12 16:40:58,939 INFO: Listing a collection in each service...
2012-03-12 16:40:59,056 ERROR: GET to http://10.32.0.5:80/user/root/ failed with status 500: {"message": "An internal error occurred.", "reference": "SCHWTNLYJIPO"}
2012-03-12 16:40:59,147 ERROR: GET to http://10.32.0.5:80/permission/object/root/ failed with status 500: {"message": "An internal error occurred.", "reference": "H5FQ3C5OOX6F"}
2012-03-12 16:40:59,187 ERROR: GET to http://10.32.0.5:80/imagelist/ failed with status 500: {"message": "An internal error occurred.", "reference": "AXJF6UE7KTLX"}
2012-03-12 16:40:59,227 ERROR: GET to http://10.32.0.5:80/instance/ failed with status 500: {"message": "An internal error occurred.", "reference": "AXJF6UMJWXN5"}
2012-03-12 16:40:59,267 ERROR: GET to http://10.32.0.5:80/proxysite/ failed with status 500: {"message": "An internal error occurred.", "reference": "AXJF6UXV3EQ2"}
2012-03-12 16:40:59,311 ERROR: GET to http://10.32.0.5:80/vethernet/ failed with status 500: {"message": "An internal error occurred.", "reference": "AXJF6UBDHK73"}
2012-03-12 16:40:59,398 ERROR: GET to http://10.32.0.5:80/seclist/ failed with status 500: {"message": "An internal error occurred.", "reference": "AXJF6UJTRIGH"}
2012-03-12 16:40:59,507 ERROR: GET to http://10.32.0.5:80/vservice/vrouterservice/ failed with status 500: {"message": "An internal error occurred.", "reference": "AXJF6UNSFTO4"}
2012-03-12 16:40:59,546 ERROR: GET to http://10.32.0.5:80/storagepool/ failed with status 500: {"message": "An internal error occurred.", "reference": "AXJF6UFDH6M7"}


What can I do to fix my cloud??

Cheers
Diego

Hi Diego,

Can you run the following to gather debug logs?

sudo nimbula-test-services
sudo nimbula-get-logs -n -f '2 hours ago'

nimbula-test-services should reproduce those internal errors, and then the log gathering process should capture the relevant information for our engineers.

Also check that your environment is compatible to running Nimbula Director with the attached checklist.

Attachments: