Derek Crudgington http://hell.jedicoder.net dacrud@gmail.com The problem I was having is trying to upload something via FTP through a web application and it wasn't uploading for some reason. The FTP daemon was receiving the commands but the file wasn't showing up in the directory. FTP log of the session trying to upload: Oct 27 14:54:26 callacctweb ftpd[25157]: [ID 165209 daemon.info] USER pinnftp Oct 27 14:54:26 callacctweb ftpd[25157]: [ID 125383 daemon.info] PASS password Oct 27 14:54:26 callacctweb ftpd[25157]: [ID 201503 daemon.info] CWD / Oct 27 14:54:26 callacctweb ftpd[25157]: [ID 313182 daemon.info] PORT Oct 27 14:54:26 callacctweb ftpd[25157]: [ID 307055 daemon.info] TYPE Image Oct 27 14:54:26 callacctweb ftpd[25157]: [ID 206289 daemon.info] STOR logo-left.jpg Nothing looks suspicious here, and even to test if it was a permission error I went ahead and made the directory +w by everyone. Still, I kept getting the same stuff, and no file showing up in the directory. So I run opensnoop (as I tell the web app to upload the file): 3555676610088 2005 Oct 27 15:05:23 0 25165 -1 2 /var/ld/ld.config /usr/sbin/in.ftpd -a -L\0 3555676610181 2005 Oct 27 15:05:23 0 25165 3 0 /lib/libsocket.so.1 /usr/sbin/in.ftpd -a -L\0 3555676610425 2005 Oct 27 15:05:23 0 25165 3 0 /lib/libnsl.so.1 /usr/sbin/in.ftpd -a -L\0 3555676610736 2005 Oct 27 15:05:23 0 25165 3 0 /lib/libpam.so.1 /usr/sbin/in.ftpd -a -L\0 3555676610925 2005 Oct 27 15:05:23 0 25165 3 0 /lib/libbsm.so.1 /usr/sbin/in.ftpd -a -L\0 3555676611205 2005 Oct 27 15:05:23 0 25165 3 0 /lib/libsendfile.so.1 /usr/sbin/in.ftpd -a -L\0 3555676611397 2005 Oct 27 15:05:23 0 25165 3 0 /usr/lib/libgss.so.1 /usr/sbin/in.ftpd -a -L\0 3555676611601 2005 Oct 27 15:05:23 0 25165 3 0 /lib/libc.so.1 /usr/sbin/in.ftpd -a -L\0 3555676612013 2005 Oct 27 15:05:23 0 25165 3 0 /lib/libcmd.so.1 /usr/sbin/in.ftpd -a -L\0 3555676612213 2005 Oct 27 15:05:23 0 25165 3 0 /lib/libmd5.so.1 /usr/sbin/in.ftpd -a -L\0 3555676612402 2005 Oct 27 15:05:23 0 25165 3 0 /lib/libsecdb.so.1 /usr/sbin/in.ftpd -a -L\0 3555676614470 2005 Oct 27 15:05:23 0 25165 3 0 /platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1 /usr/sbin/in.ftpd -a -L\0 3555676614737 2005 Oct 27 15:05:23 0 25165 3 0 /dev/conslog /usr/sbin/in.ftpd -a -L\0 3555676615048 2005 Oct 27 15:05:23 0 25165 2 0 /dev/null /usr/sbin/in.ftpd -a -L\0 3555676615178 2005 Oct 27 15:05:23 0 25165 4 0 /etc/ftpd/ftpservers /usr/sbin/in.ftpd -a -L\0 3555676615394 2005 Oct 27 15:05:23 0 25165 4 0 /etc/ftpd/ftpaccess /usr/sbin/in.ftpd -a -L\0 3555676621611 2005 Oct 27 15:05:23 0 25165 4 0 /etc/netconfig /usr/sbin/in.ftpd -a -L\0 3555676621896 2005 Oct 27 15:05:23 0 25165 4 0 /dev/udp /usr/sbin/in.ftpd -a -L\0 3555676622083 2005 Oct 27 15:05:23 0 25165 4 0 /etc/nsswitch.conf /usr/sbin/in.ftpd -a -L\0 3555676622717 2005 Oct 27 15:05:23 0 25165 4 0 /lib/nss_files.so.1 /usr/sbin/in.ftpd -a -L\0 3555676623097 2005 Oct 27 15:05:23 0 25165 4 0 /etc/services /usr/sbin/in.ftpd -a -L\0 3555676623385 2005 Oct 27 15:05:23 0 25165 4 0 /var/run/name_service_door /usr/sbin/in.ftpd -a -L\0 3555676623700 2005 Oct 27 15:05:23 0 25165 5 0 /etc/ftpd/ftpconversions /usr/sbin/in.ftpd -a -L\0 3555676623868 2005 Oct 27 15:05:23 0 25165 -1 2 /etc/ftpd/banner.msg /usr/sbin/in.ftpd -a -L\0 3555676624802 2005 Oct 27 15:05:23 0 25165 5 0 /etc/ftpd/ftphosts /usr/sbin/in.ftpd -a -L\0 3555676624967 2005 Oct 27 15:05:23 0 25165 5 0 /etc/shadow /usr/sbin/in.ftpd -a -L\0 3555676625076 2005 Oct 27 15:05:23 0 25165 -1 2 /etc/shells /usr/sbin/in.ftpd -a -L\0 3555676625110 2005 Oct 27 15:05:23 0 25165 5 0 /etc/ftpd/ftpusers /usr/sbin/in.ftpd -a -L\0 3555676625159 2005 Oct 27 15:05:23 0 25165 -1 2 /etc/ftpusers /usr/sbin/in.ftpd -a -L\0 3555676625261 2005 Oct 27 15:05:23 0 25165 5 0 /var/run/ftp.pids-realusers /usr/sbin/in.ftpd -a -L\0 3555676625499 2005 Oct 27 15:05:23 0 25165 6 0 /usr/share/lib/zoneinfo/US/Central /usr/sbin/in.ftpd -a -L\0 3555676625674 2005 Oct 27 15:05:23 0 25165 6 0 /var/run/syslog_door /usr/sbin/in.ftpd -a -L\0 3555676625873 2005 Oct 27 15:05:23 0 25165 6 0 /var/run/syslog_door /usr/sbin/in.ftpd -a -L\0 3555676626002 2005 Oct 27 15:05:23 0 25165 -1 2 /etc/pam_debug /usr/sbin/in.ftpd -a -L\0 3555676626071 2005 Oct 27 15:05:23 0 25165 6 0 /etc/pam.conf /usr/sbin/in.ftpd -a -L\0 3555676626432 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/security/pam_authtok_get.so.1 /usr/sbin/in.ftpd -a -L\0 3555676626665 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/passwdutil.so.1 /usr/sbin/in.ftpd -a -L\0 3555676626912 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/libsldap.so.1 /usr/sbin/in.ftpd -a -L\0 3555676627230 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/libldap.so.5 /usr/sbin/in.ftpd -a -L\0 3555676627558 2005 Oct 27 15:05:23 0 25165 6 0 /lib/libdoor.so.1 /usr/sbin/in.ftpd -a -L\0 3555676627786 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/libsasl.so.1 /usr/sbin/in.ftpd -a -L\0 3555676628099 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/mps/libnspr4.so /usr/sbin/in.ftpd -a -L\0 3555676628353 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/mps/libplc4.so /usr/sbin/in.ftpd -a -L\0 3555676628546 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/mps/libnss3.so /usr/sbin/in.ftpd -a -L\0 3555676628832 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/mps/libssl3.so /usr/sbin/in.ftpd -a -L\0 3555676629066 2005 Oct 27 15:05:23 0 25165 6 0 /lib/libpthread.so.1 /usr/sbin/in.ftpd -a -L\0 3555676629205 2005 Oct 27 15:05:23 0 25165 6 0 /lib/libthread.so.1 /usr/sbin/in.ftpd -a -L\0 3555676629379 2005 Oct 27 15:05:23 0 25165 6 0 /lib/librt.so.1 /usr/sbin/in.ftpd -a -L\0 3555676629590 2005 Oct 27 15:05:23 0 25165 6 0 /lib/libdl.so.1 /usr/sbin/in.ftpd -a -L\0 3555676629726 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/mps/libsoftokn3.so /usr/sbin/in.ftpd -a -L\0 3555676629992 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/mps/libplds4.so /usr/sbin/in.ftpd -a -L\0 3555676533646 2005 Oct 27 15:05:23 60001 24924 -1 2 /usr/apache/pinnacleweb/.htaccess /usr/apache/bin/httpd -DSSL\0 3555676533700 2005 Oct 27 15:05:23 60001 24924 -1 2 /usr/apache/pinnacleweb/web/.htaccess /usr/apache/bin/httpd -DSSL\0 3555676533746 2005 Oct 27 15:05:23 60001 24924 -1 2 /usr/apache/pinnacleweb/web/app/.htaccess /usr/apache/bin/httpd -DSSL\0 3555676534078 2005 Oct 27 15:05:23 60001 24924 7 0 /var/apache/logs/owalog.log /usr/apache/bin/httpd -DSSL\0 3555676534131 2005 Oct 27 15:05:23 60001 24924 7 0 /var/apache/logs/owalog.log /usr/apache/bin/httpd -DSSL\0 3555676534199 2005 Oct 27 15:05:23 60001 24924 7 0 /var/apache/logs/owalog.log /usr/apache/bin/httpd -DSSL\0 3555676534245 2005 Oct 27 15:05:23 60001 24924 7 0 /var/apache/logs/owalog.log /usr/apache/bin/httpd -DSSL\0 3555676534292 2005 Oct 27 15:05:23 60001 24924 7 0 /var/apache/logs/owalog.log /usr/apache/bin/httpd -DSSL\0 3555676584871 2005 Oct 27 15:05:23 0 23893 30 0 /system/contract/process/latest /usr/lib/inet/inetd start\0 3555676585030 2005 Oct 27 15:05:23 0 23893 30 0 /proc/25165/psinfo /usr/lib/inet/inetd start\0 3555676587387 2005 Oct 27 15:05:23 0 18224 13 0 /etc/svc/volatile/svc_nonpersist.db-journal /lib/svc/bin/svc.configd\03555676587430 2005 Oct 27 15:05:23 0 18224 14 0 /etc/svc/volatile /lib/svc/bin/svc.configd\0 3555676587865 2005 Oct 27 15:05:23 0 18224 17 0 /etc/svc/volatile/sqlite_9C3PWfLqAwfyW8l /lib/svc/bin/svc.configd\0 3555676590820 2005 Oct 27 15:05:23 0 18224 13 0 /etc/svc/volatile/svc_nonpersist.db-journal /lib/svc/bin/svc.configd\03555676590857 2005 Oct 27 15:05:23 0 18224 14 0 /etc/svc/volatile /lib/svc/bin/svc.configd\0 3555676591223 2005 Oct 27 15:05:23 0 18224 17 0 /etc/svc/volatile/sqlite_KPSalwKdewEwutR /lib/svc/bin/svc.configd\0 3555676661709 2005 Oct 27 15:05:23 155 25165 -1 2 .message /usr/sbin/in.ftpd -a -L\0 3555676661744 2005 Oct 27 15:05:23 155 25165 7 0 . /usr/sbin/in.ftpd -a -L\0 3555676674088 2005 Oct 27 15:05:23 155 25165 7 0 /var/run/syslog_door /usr/sbin/in.ftpd -a -L\0 3555676674655 2005 Oct 27 15:05:23 155 25165 7 0 /var/run/syslog_door /usr/sbin/in.ftpd -a -L\0 3555676675148 2005 Oct 27 15:05:23 155 25165 7 0 /var/run/syslog_door /usr/sbin/in.ftpd -a -L\0 3555676675186 2005 Oct 27 15:05:23 155 25165 7 0 . /usr/sbin/in.ftpd -a -L\0 3555676675239 2005 Oct 27 15:05:23 155 25165 7 0 . /usr/sbin/in.ftpd -a -L\0 3555676675271 2005 Oct 27 15:05:23 155 25165 7 0 . /usr/sbin/in.ftpd -a -L\0 3555676675396 2005 Oct 27 15:05:23 155 25165 7 0 . /usr/sbin/in.ftpd -a -L\0 3555676675514 2005 Oct 27 15:05:23 155 25165 -1 13 logo-left.jpg /usr/sbin/in.ftpd -a -L\0 3555676681446 2005 Oct 27 15:05:23 0 18224 13 0 /etc/svc/volatile/svc_nonpersist.db-journal /lib/svc/bin/svc.configd\03555676681500 2005 Oct 27 15:05:23 0 18224 14 0 /etc/svc/volatile /lib/svc/bin/svc.configd\0 3555676681949 2005 Oct 27 15:05:23 0 18224 17 0 /etc/svc/volatile/sqlite_3bPs9j3k6YGQuOq /lib/svc/bin/svc.configd\0 3555676685019 2005 Oct 27 15:05:23 0 18224 13 0 /etc/svc/volatile/svc_nonpersist.db-journal /lib/svc/bin/svc.configd\03555676685056 2005 Oct 27 15:05:23 0 18224 14 0 /etc/svc/volatile /lib/svc/bin/svc.configd\0 3555676685422 2005 Oct 27 15:05:23 0 18224 17 0 /etc/svc/volatile/sqlite_fcgHWsRuwiXRABD /lib/svc/bin/svc.configd\0 3555676686749 2005 Oct 27 15:05:23 0 23893 29 0 /system/contract/process/5264/ctl /usr/lib/inet/inetd start\0 3555676632922 2005 Oct 27 15:05:23 0 25165 6 0 /lib/libaio.so.1 /usr/sbin/in.ftpd -a -L\0 3555676633753 2005 Oct 27 15:05:23 0 25165 6 0 /platform/SUNW,Sun-Fire-V240/lib/libmd5_psr.so.1 /usr/sbin/in.ftpd -a -L\0 3555676634324 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/mps/cpu/sparcv8plus/libnspr_flt4.so /usr/sbin/in.ftpd -a -L\03555676635987 2005 Oct 27 15:05:23 0 25165 6 0 /lib/libmp.so.2 /usr/sbin/in.ftpd -a -L\0 3555676636242 2005 Oct 27 15:05:23 0 25165 6 0 /lib/libscf.so.1 /usr/sbin/in.ftpd -a -L\0 3555676636573 2005 Oct 27 15:05:23 0 25165 6 0 /lib/libuutil.so.1 /usr/sbin/in.ftpd -a -L\0 3555676637603 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/security/pam_dhkeys.so.1 /usr/sbin/in.ftpd -a -L\0 3555676638056 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/security/pam_unix_cred.so.1 /usr/sbin/in.ftpd -a -L\0 3555676638294 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/libproject.so.1 /usr/sbin/in.ftpd -a -L\0 3555676638516 2005 Oct 27 15:05:23 0 25165 6 0 /lib/libproc.so.1 /usr/sbin/in.ftpd -a -L\0 3555676638733 2005 Oct 27 15:05:23 0 25165 6 0 /lib/librtld_db.so.1 /usr/sbin/in.ftpd -a -L\0 3555676638942 2005 Oct 27 15:05:23 0 25165 6 0 /lib/libelf.so.1 /usr/sbin/in.ftpd -a -L\0 3555676639155 2005 Oct 27 15:05:23 0 25165 6 0 /lib/libctf.so.1 /usr/sbin/in.ftpd -a -L\0 3555676639697 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/security/pam_unix_auth.so.1 /usr/sbin/in.ftpd -a -L\0 3555676640126 2005 Oct 27 15:05:23 0 25165 6 0 /etc/nsswitch.conf /usr/sbin/in.ftpd -a -L\0 3555676640282 2005 Oct 27 15:05:23 0 25165 6 0 /etc/shadow /usr/sbin/in.ftpd -a -L\0 3555676640673 2005 Oct 27 15:05:23 0 25165 6 0 /etc/shadow /usr/sbin/in.ftpd -a -L\0 3555676640810 2005 Oct 27 15:05:23 0 25165 6 0 /etc/passwd /usr/sbin/in.ftpd -a -L\0 3555676640883 2005 Oct 27 15:05:23 0 25165 6 0 /etc/shadow /usr/sbin/in.ftpd -a -L\0 3555676641404 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/security/pam_roles.so.1 /usr/sbin/in.ftpd -a -L\0 3555676641728 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/security/pam_unix_account.so.1 /usr/sbin/in.ftpd -a -L\0 3555676642098 2005 Oct 27 15:05:23 0 18260 5 0 /etc/user_attr /usr/sbin/nscd\0 3555676642286 2005 Oct 27 15:05:23 0 25165 6 0 /etc/shadow /usr/sbin/in.ftpd -a -L\0 3555676642376 2005 Oct 27 15:05:23 0 25165 6 0 /var/adm/lastlog /usr/sbin/in.ftpd -a -L\0 3555676642462 2005 Oct 27 15:05:23 0 25165 6 0 /etc/default/login /usr/sbin/in.ftpd -a -L\0 3555676642660 2005 Oct 27 15:05:23 0 25165 6 0 /usr/lib/security/pam_unix_session.so.1 /usr/sbin/in.ftpd -a -L\0 3555676642959 2005 Oct 27 15:05:23 0 25165 6 0 /var/adm/lastlog /usr/sbin/in.ftpd -a -L\0 3555676658272 2005 Oct 27 15:05:23 0 25165 6 0 /etc/shadow /usr/sbin/in.ftpd -a -L\0 3555676658591 2005 Oct 27 15:05:23 0 25165 6 0 /etc/project /usr/sbin/in.ftpd -a -L\0 3555676658765 2005 Oct 27 15:05:23 0 25165 6 0 /etc/project /usr/sbin/in.ftpd -a -L\0 3555676658809 2005 Oct 27 15:05:23 0 25165 6 0 /etc/project /usr/sbin/in.ftpd -a -L\0 3555676658944 2005 Oct 27 15:05:23 0 25165 6 0 /etc/project /usr/sbin/in.ftpd -a -L\0 3555676659038 2005 Oct 27 15:05:23 0 25165 6 0 /etc/project /usr/sbin/in.ftpd -a -L\0 3555676659352 2005 Oct 27 15:05:23 0 25165 6 0 /etc/security/policy.conf /usr/sbin/in.ftpd -a -L\0 3555676659657 2005 Oct 27 15:05:23 0 25165 6 0 /etc/default/nss /usr/sbin/in.ftpd -a -L\0 3555676659978 2005 Oct 27 15:05:23 0 25165 6 0 /etc/group /usr/sbin/in.ftpd -a -L\0 3555676660095 2005 Oct 27 15:05:23 0 25165 6 0 /var/adm/wtmpx /usr/sbin/in.ftpd -a -L\0 3555676660331 2005 Oct 27 15:05:23 0 25165 -1 2 /etc/ftpd/welcome.msg /usr/sbin/in.ftpd -a -L\0 3555676660361 2005 Oct 27 15:05:23 0 25165 7 0 . /usr/sbin/in.ftpd -a -L\0 3555676660457 2005 Oct 27 15:05:23 0 25165 -1 2 .message /usr/sbin/in.ftpd -a -L\0 3555676660476 2005 Oct 27 15:05:23 0 25165 7 0 . /usr/sbin/in.ftpd -a -L\0 3555676660732 2005 Oct 27 15:05:23 0 25165 7 0 . /usr/sbin/in.ftpd -a -L\0 3555676661510 2005 Oct 27 15:05:23 155 25165 7 0 /var/run/syslog_door /usr/sbin/in.ftpd -a -L\0 3555676661583 2005 Oct 27 15:05:23 155 25165 7 0 . /usr/sbin/in.ftpd -a -L\0 3555676748932 2005 Oct 27 15:05:23 60001 24924 7 0 /var/apache/logs/owalog.log /usr/apache/bin/httpd -DSSL\0 3555676749010 2005 Oct 27 15:05:23 60001 24924 7 0 /var/apache/logs/owalog.log /usr/apache/bin/httpd -DSSL\0 3555676749057 2005 Oct 27 15:05:23 60001 24924 7 0 /var/apache/logs/owalog.log /usr/apache/bin/httpd -DSSL\0 3555676835176 2005 Oct 27 15:05:23 60001 24924 -1 2 /usr/apache/pinnacleweb/.htaccess /usr/apache/bin/httpd -DSSL\0 3555676835233 2005 Oct 27 15:05:23 60001 24924 -1 2 /usr/apache/pinnacleweb/web/.htaccess /usr/apache/bin/httpd -DSSL\0 3555676835279 2005 Oct 27 15:05:23 60001 24924 -1 2 /usr/apache/pinnacleweb/web/app/.htaccess /usr/apache/bin/httpd -DSSL\ 0 3555676835329 2005 Oct 27 15:05:23 60001 24924 -1 2 /usr/apache/pinnacleweb/web/app/uploads/.htaccess /usr/apache/bin/httpd -DSSL\0 3555677450168 2005 Oct 27 15:05:24 0 99 4 0 /devices/pci@1c,600000/scsi@2:devctl /usr/lib/picl/picld\0 3555677450299 2005 Oct 27 15:05:24 0 99 4 0 /devices/pci@1c,600000/scsi@2:devctl /usr/lib/picl/picld\0 3555677450359 2005 Oct 27 15:05:24 0 99 4 0 /devices/pci@1c,600000/scsi@2:devctl /usr/lib/picl/picld\0 3555677450414 2005 Oct 27 15:05:24 0 99 4 0 /devices/pci@1c,600000/scsi@2:devctl /usr/lib/picl/picld\0 ^C What were looking for here is all of the '-1' FD values. This indicates there was an error somewhere. Looking at all of them, there's only one that really sticks out, and its the name of the image I was trying to upload, logo-left.jpg It has an error number of 13, lets see what this means: root@callacct:~/DTraceToolkit-0.83# grep 13 /usr/include/sys/errno.h #define EACCES 13 /* Permission denied */ #define ECONNABORTED 130 /* Software caused connection abort */ #define ECONNRESET 131 /* Connection reset by peer */ #define ENOBUFS 132 /* No buffer space available */ #define EISCONN 133 /* Socket is already connected */ #define ENOTCONN 134 /* Socket is not connected */ /* XENIX has 135 - 142 */ It is showing I'm getting a permission denied. This makes more sense if you look at the FTP log it shows it issuing "CWD /" then trying to upload a file which means its changing to the root (/) directory first then uploading the file and failing with the permission denied. The root directory doesn't have permission to let anyone upload there and the FTP web app shouldn't be issuing a 'cd /' and then trying to upload the file.