Norman Meister

I am using "jxl-2.6.9". The "excel_original.xls" on processing column B "bis", the values "00:15:00" is read as 12:15, next value "00:30:00" as 12:30 and so. This is wrong, the time value should have been read as it is. "00:15:00" as "00:15:00" and so on.

The "excel_original.xls" i received directly from the supplier as attached in the mailbox. The system picks the attachment and processes it incorrectly. But when I "locally open and save" the file and manually process it, the system treats it correctly. Something changes in the format of the file when I locally save it without even making any change.

The changed file version is "excel_changed.xls". The file size increases when I locally save it. The contents are exactly the same in both the files.

Can someone suggest the reason for this behavior?

thanks.

I have asked the same question http://www.javaprogrammingforums.com/whats-wrong-my-code/45615-excel-file-processing-issue-java.html#post183909 at this forum as well, but no answer. Also you will find on this other forum page the excel files attached. I could not attach them here.

import java.io.FileInputStream; import java.io.IOException; import jxl.LabelCell; import jxl.Sheet; import jxl.Workbook; import jxl.read.biff.BiffException; public class ProcessXLS { public static void main(String args[]) throws BiffException, IOException { String FilePath = "/excel_original.xls"; // String FilePath = "/excel_changed.xls"; FileInputStream fs = new FileInputStream(FilePath); Workbook wb = Workbook.getWorkbook(fs); Sheet sh = wb.getSheet("Sheet0"); LabelCell l = sh.findLabelCell("bis"); System.out.println("Content : " + l.getContents()); for (int i = 14; i < 110; i++) { System.out.println(sh.getCell(1, i).getContents()); System.out.println(sh.getCell(1, i).getType()); } } }

any help?

Sorry i forgot to mention Java version. It is 6.

hi all,

I have the following specs:

System Memory : 8,00 GB.
32bit Windows server
JVM Xmx : 1024m
Glassfish Server 3.1

My application keeps crashing (page not accessible HTTP status 500), and on my checking the Heap space, the permgen size is generally always full or close to full.

I am trying to analyse the application for the memory leaks, but i was also wondering if someone could recommend me better/optimized JVM parameters' settings on the machine that might help.

Tim Holloway wrote:POP isn't a very powerful protocol compared to IMAP. If I'm not mistaken, it doesn't have folders or a query-based retrieval capability. Personally, I'd run an "nmap" on the POP server and see if IMAP/IMAPS is truly not an option, since a lot of the popular email servers these days support multiple protocols. And if IMAP or IMAPS is available, lean on them. It's to their own benefit ultimately.

Every email should have a Message-Id header in it. When you download an email, keep a database of these Message-Ids and only forward the ones you haven't yet added to the database. That's less resource-intensive than keeping the entire email on file. And the Message-ID is designed to be unique, just like UUID/GUIDs are - perhaps moreso, since UUIDs and GUIDs are hash values and the email ID may be 100% unique.

Thanks for the pointer, Tim. Will look into that.

But how about deleting the emails from the server once they are forwarded to another email account? This way when the Java util checks the source mail account next time, it will only find new emails because older emails have already been deleted in the last run.

I also came across this thread: http://alvinalexander.com/java/javamail-search-unseen-unread-messages-mailbox-pop3

Regarding IMAP, unfortunately, I confirmed and the supplier said it is not supported.

Thanks.

Rob Spoor wrote:If the supplier has given you port number 110, that probably means that it's a POP server. Trying to use IMAP won't work then. Try asking the supplier if they support IMAP as well.

Regarding "new" status and POP, that should be maintained by the client. The client you're writing should keep track of (and store) which emails have already been encountered and which not. But yeah, if IMAP is available that would be better.

Thanks Rob for your reply. The supplier only provides POP3.

My scenario is that I have a scheduled java utility that checks the mailbox every 5 minutes. If there is a new email that I need to "forward" to another mailbox hosted on another mail server using SMTP. This other mailbox is checked by a local user using outlook. At this point, I have a problem since the application is unable to check if there is any "new" email in the INBOX. It forwards all the available set of emails in the inbox be it new or old to the other email address.

You suggested that the application should keep track of (and store) which emails have already been encountered and which not. What could be the standard criteria used for keeping track of already forwarded/old emails at the client side? Storing emails is not required and not in the scope of the client application.

Another option, I can think of is "moving" all those emails at the source mail account (which are forwarded) to another "folder". Next time, when the utitlity would check the inbox, it would it would only find any "new" email in the inbox if any. Do you think, this approach makes sense, and if yes, does POP3 api provide moving option or in other words, would that be a straight forward solution to implement in the java pop3 client?

many thanks.

hi all,

I need to process emails from a mailbox account.

The supplier of the email box has provided me with the host name of the server and stated the port number as 110. Also the supplier has given me with the access credentials.

My question is about the email reader client that I need to implement in Java. As per google search, 110 is the default port for POP3 interface, but can I implement the Java IMAP email client using port 110 instead?

Or in other words, is client development dependent on the type of protocol the email host supports?

Since the supplier of the email host has stated port 110, does that really mean that i can only use pop3 client to communicate with the server or can I also implement and use IMAP client that connects with the server on port 110?`

I see many advantages of using IMAP interface than POP3, specially in the context of already processed email. I only need to process emails which are new, and that can only be checked within IMAP (as per my current understanding). I could not find such check whether the email is new or not supported within pop3 protocol.

many thanks.

Hi All,

We have the following 2 similar excel files, available at Here

File_with_Issue.xls
File_Without_Issue.xls

The first "File_with_Issue.xls" file we received from the supplier and when we process it as it is with our application, we get ArrayIndexOutOfBoundException.

The same file when we locally OPEN in Excel and SAVE it locally and then give to the application for processing, it processes the file well and without any issues.

The exception comes when the last Quarter Bis value 00:00:00 is processed.

In case of first file, we get the application output value as "12:00" against the 00:00:00, whereas, in-case of locally saved file, we get the returned value as "0:00" from the application.

We have a condition in the code to read the positions. In case of "File_Without_Issue.xls" the condition is true and the loop execution is stopped. But in case of "File_with_Issue.xls", even after reaching the last quarter, the logic will try to fetch the values from an index outside of range. So, the application throws ArrayIndexOutOfBoundException.


do { endPosition = getCellContents(sheet, endPositionRef); System.out.println("endPosition:"+endPosition); String quantityStart = getCellContents(sheet, quantityStartRef); Timestamp quantityStartTimeStamp = addTime(scheduleDate, quantityStart, XLS_TIME_ZONE); BigDecimal quantity = parsePositiveNumber(getCellContents( sheet, quantityRef)); fixSummerWinterTimeSwitch(quantityStartTimeStamp, positionList); SQL_ETDSchedulePos etdPos = new SQL_ETDSchedulePos( quantityStartTimeStamp, quantity); positionList.add(etdPos); quantityStartRef.nextRow(); endPositionRef.nextRow(); quantityRef.nextRow(); } while (!(endPosition == null || endPosition.equals("") || endPosition.equalsIgnoreCase("0:00") || endPosition.equalsIgnoreCase("00:00") || endPosition .equalsIgnoreCase("24:00")));

In my opinion, the problem probably is due to different excel version used to prepare/save the file?

I have MS Excel 2010, but when i saved as from the original file "File_with_Issue.xls", the default file type used was Excel 97-2003 Workbook. And then the new file "File_Without_Issue.xls" got processed correctly.

Also point to note is the file size. The original file size is: 143 kb, whereas the newly generated file size is 189 kb.

Can someone give any pointers as to what could be the actual problem?

It does not make sense due to obvious reason to first locally save the file using local excel version and then have it processed. The original file should be processed directly.

thanks.

Let's have a look at the JVM memory utilization stats with graph generated using PSI-Probe (attached below).

The strange thing is the PS Old gen memory utilization.

The JVM settings in Tomcat are as follows:

***

# discourage address map swapping by setting Xms and Xmx to the same value
export CATALINA_OPTS="$CATALINA_OPTS -Xms4096m"
export CATALINA_OPTS="$CATALINA_OPTS -Xmx4096m"

# Increase maximum perm size for web base applications to 4x the default amount
# http://wiki.apache.org/tomcat/FAQ/Memoryhttp://wiki.apache.org/tomcat/FAQ/Memory
export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxPermSize=512m"

# Reset the default stack size for threads to a lower value (by 1/10th original)
# By default this can be anywhere between 512k -> 1024k depending on x32 or x64
# bit Java version.
# http://www.springsource.com/files/uploads/tomcat/tomcatx-large-scale-deployments.pdf
# http://www.oracle.com/technetwork/java/hotspotfaq-138619.html
# Note - JDK 7 on RHEL needs minimum -Xss256k
#export CATALINA_OPTS="$CATALINA_OPTS -Xss192k"

# Oracle Java as default, uses the serial garbage collector on the
# Full Tenured heap. The Young space is collected in parallel, but the
# Tenured is not. This means that at a time of load if a full collection
# event occurs, since the event is a 'stop-the-world' serial event then
# all application threads other than the garbage collector thread are
# taken off the CPU. This can have severe consequences if requests continue
# to accrue during these 'outage' periods. (specifically webservices, webapps)
# [Also enables adaptive sizing automatically]
export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelGC"

# This is interpreted as a hint to the garbage collector that pause times
# of <nnn> milliseconds or less are desired. The garbage collector will
# adjust the Java heap size and other garbage collection related parameters
# in an attempt to keep garbage collection pauses shorter than <nnn> milliseconds.
# http://java.sun.com/docs/hotspot/gc5.0/ergo5.html
export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=1500"

# A hint to the virtual machine that it.s desirable that not more than:
# 1 / (1 + GCTimeRation) of the application execution time be spent in
# the garbage collector.
# http://themindstorms.wordpress.com/2009/01/21/advanced-jvm-tuning-for-low-pause/
export CATALINA_OPTS="$CATALINA_OPTS -XX:GCTimeRatio=9"

# The hotspot server JVM has specific code-path optimizations
# which yield an approximate 10% gain over the client version.
export CATALINA_OPTS="$CATALINA_OPTS -server"

# Disable remote (distributed) garbage collection by Java clients
# and remove ability for applications to call explicit GC collection
export CATALINA_OPTS="$CATALINA_OPTS -XX:+DisableExplicitGC"

# Uncomment to enable jconsole remote connection management.
# export CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote
# -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9011
# -Dcom.sun.management.jmxremote.local.only=false
# -Dcom.sun.management.jmxremote.authenticate=false"

# Check for application specific parameters at startup
if [ -r "$CATALINA_BASE/bin/appenv.sh" ]; then
. "$CATALINA_BASE/bin/appenv.sh"
fi

echo "Using CATALINA_OPTS:"
for arg in $CATALINA_OPTS
do
echo ">> " $arg
done
echo ""

echo "Using JAVA_OPTS:"
for arg in $JAVA_OPTS
do
echo ">> " $arg
done
echo "_______________________________________________"
echo ""

***

Tim Holloway wrote:At the OS level, you might want to set up an ongoing vmstat whose results are collected to a file and then plotted with gnuplot or something like that (assuming you'd like graphs instead of just raw numbers). VMStat can help detect "spikes" in resource demands over time, where "top" displays only the demand of the instant.

This looks like it might help you understand how vmstat is used: http://www.admin-magazine.com/HPC/Articles/Stat-like-Tools-for-Admins

At the JVM level, try running the jprofile tool to see how it's using memory internally. Considering all the grief, it might be worth spending some money on some commercial analysis tools.

Attached are some stats. I will also try to get vmstat.


****

ABCD2:/webdienste/appserv-jsp/tomcat/ABCD2.com/current # ps aux | sort -k 5n
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 10 0.0 0.0 0 0 ? S Jan28 3:00 [ksoftirqd/1]
root 11 0.0 0.0 0 0 ? S Jan28 19:59 [kworker/0:1]
root 1154 0.0 0.0 0 0 ? S Jan28 0:00 [kauditd]
root 1177 0.0 0.0 0 0 ? SN Jan28 5:43 [kipmi0]
root 1193 0.0 0.0 0 0 ? S< Jan28 0:00 [kmpathd]
root 1194 0.0 0.0 0 0 ? S< Jan28 0:00 [kmpath_handlerd]
root 12 0.0 0.0 0 0 ? S Jan28 0:51 [watchdog/1]
root 12112 0.0 0.0 0 0 ? S Mar26 2:17 [kworker/7:2]
root 13 0.0 0.0 0 0 ? S Jan28 0:02 [migration/2]
root 1340 0.0 0.0 0 0 ? S Jan28 0:00 [kjournald]
root 1341 0.0 0.0 0 0 ? S Jan28 0:00 [kjournald]
root 1342 0.0 0.0 0 0 ? S Jan28 0:52 [kjournald]
root 1343 0.0 0.0 0 0 ? S Jan28 6:16 [kjournald]
root 1344 0.0 0.0 0 0 ? S Jan28 0:00 [kjournald]
root 1345 0.0 0.0 0 0 ? S Jan28 0:00 [kjournald]
root 1346 0.0 0.0 0 0 ? S Jan28 0:00 [kjournald]
root 1347 0.0 0.0 0 0 ? S Jan28 0:19 [kjournald]
root 139 0.0 0.0 0 0 ? S< Jan28 0:34 [lpfc_worker_0]
root 140 0.0 0.0 0 0 ? S Jan28 0:00 [scsi_eh_0]
root 141 0.0 0.0 0 0 ? S< Jan28 0:00 [scsi_wq_0]
root 142 0.0 0.0 0 0 ? S Jan28 17:08 [kworker/0:2]
root 143 0.0 0.0 0 0 ? S Jan28 1:09 [kworker/13:1]
root 144 0.0 0.0 0 0 ? S Jan28 1:15 [kworker/14:1]
root 145 0.0 0.0 0 0 ? S< Jan28 0:29 [lpfc_worker_1]
root 146 0.0 0.0 0 0 ? S Jan28 0:00 [scsi_eh_1]
root 147 0.0 0.0 0 0 ? S< Jan28 0:00 [scsi_wq_1]
root 148 0.0 0.0 0 0 ? S Jan28 2:03 [kworker/1:1]
root 149 0.0 0.0 0 0 ? S Jan28 1:56 [kworker/2:1]
root 15 0.0 0.0 0 0 ? S Jan28 1:56 [ksoftirqd/2]
root 155 0.0 0.0 0 0 ? S Jan28 2:02 [kworker/4:1]
root 156 0.0 0.0 0 0 ? S Jan28 2:22 [kworker/5:1]
root 157 0.0 0.0 0 0 ? S Jan28 1:21 [kworker/8:1]
root 158 0.0 0.0 0 0 ? S Jan28 1:27 [kworker/10:1]
root 160 0.0 0.0 0 0 ? S Jan28 1:19 [kworker/15:1]
root 16 0.0 0.0 0 0 ? S Jan28 0:46 [watchdog/2]
root 162 0.0 0.0 0 0 ? S Jan28 1:53 [kworker/3:1]
root 163 0.0 0.0 0 0 ? S Jan28 0:00 [scsi_eh_2]
root 164 0.0 0.0 0 0 ? S Jan28 0:02 [hpsa]
root 17 0.0 0.0 0 0 ? S Jan28 0:03 [migration/3]
root 18 0.0 0.0 0 0 ? S Jan28 2:25 [kworker/3:0]
root 189 0.0 0.0 0 0 ? S Jan28 1:16 [kworker/9:1]
root 190 0.0 0.0 0 0 ? S Jan28 1:25 [kworker/12:1]
root 19 0.0 0.0 0 0 ? S Jan28 2:53 [ksoftirqd/3]
root 1927 0.0 0.0 0 0 ? S Jan28 1:11 [flush-253:5]
root 1928 0.0 0.0 0 0 ? S Jan28 6:48 [flush-253:6]
root 199 0.0 0.0 0 0 ? S< Jan28 0:00 [kmpath_aluad]
root 20 0.0 0.0 0 0 ? S Jan28 0:47 [watchdog/3]
root 2 0.0 0.0 0 0 ? S Jan28 0:00 [kthreadd]
root 21 0.0 0.0 0 0 ? S Jan28 0:02 [migration/4]
root 214 0.0 0.0 0 0 ? S< Jan28 0:00 [kmpath_rdacd]
root 22 0.0 0.0 0 0 ? S Jan28 2:21 [kworker/4:0]
root 23 0.0 0.0 0 0 ? S Jan28 3:14 [ksoftirqd/4]
root 23243 0.0 0.0 0 0 ? S Aug03 0:26 [kworker/2:2]
root 234 0.0 0.0 0 0 ? S Jan28 1:12 [kworker/11:1]
root 24 0.0 0.0 0 0 ? S Jan28 0:48 [watchdog/4]
root 25 0.0 0.0 0 0 ? S Jan28 0:03 [migration/5]
root 26 0.0 0.0 0 0 ? S Jan28 2:21 [kworker/5:0]
root 27 0.0 0.0 0 0 ? S Jan28 3:30 [ksoftirqd/5]
root 28 0.0 0.0 0 0 ? S Jan28 0:48 [watchdog/5]
root 29 0.0 0.0 0 0 ? S Jan28 0:02 [migration/6]
root 3 0.0 0.0 0 0 ? S Jan28 4:11 [ksoftirqd/0]
root 31 0.0 0.0 0 0 ? S Jan28 5:47 [ksoftirqd/6]
root 32 0.0 0.0 0 0 ? S Jan28 0:47 [watchdog/6]
root 33 0.0 0.0 0 0 ? S Jan28 0:03 [migration/7]
root 34 0.0 0.0 0 0 ? S Jan28 3:04 [kworker/7:0]
root 34731 0.0 0.0 0 0 ? S 12:27 0:00 [flush-253:3]
root 35 0.0 0.0 0 0 ? S Jan28 6:29 [ksoftirqd/7]
root 3557 0.0 0.0 0 0 ? S< Jan28 0:00 [bond0]
root 36 0.0 0.0 0 0 ? S Jan28 0:47 [watchdog/7]
root 366 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/u:1]
root 37 0.0 0.0 0 0 ? S Jan28 0:58 [migration/8]
root 38 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/8:0]
root 39 0.0 0.0 0 0 ? S Jan28 0:41 [ksoftirqd/8]
root 40 0.0 0.0 0 0 ? S Jan28 0:45 [watchdog/8]
root 405 0.0 0.0 0 0 ? S Jan28 0:00 [khubd]
root 41 0.0 0.0 0 0 ? S Jan28 0:59 [migration/9]
root 42 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/9:0]
root 43 0.0 0.0 0 0 ? S Jan28 0:44 [ksoftirqd/9]
root 44 0.0 0.0 0 0 ? S Jan28 0:46 [watchdog/9]
root 45 0.0 0.0 0 0 ? S Jan28 0:58 [migration/10]
root 46 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/10:0]
root 47 0.0 0.0 0 0 ? S Jan28 0:41 [ksoftirqd/10]
root 48 0.0 0.0 0 0 ? S Jan28 0:46 [watchdog/10]
root 49 0.0 0.0 0 0 ? S Jan28 0:59 [migration/11]
root 50 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/11:0]
root 5 0.0 0.0 0 0 ? S Jan28 19:29 [kworker/u:0]
root 51 0.0 0.0 0 0 ? S Jan28 0:33 [ksoftirqd/11]
root 511 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 513 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 515 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 518 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 52 0.0 0.0 0 0 ? S Jan28 0:49 [watchdog/11]
root 521 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 525 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 529 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 53 0.0 0.0 0 0 ? S Jan28 0:59 [migration/12]
root 533 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 537 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 54 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/12:0]
root 55 0.0 0.0 0 0 ? S Jan28 0:44 [ksoftirqd/12]
root 56 0.0 0.0 0 0 ? S Jan28 0:51 [watchdog/12]
root 57 0.0 0.0 0 0 ? S Jan28 1:00 [migration/13]
root 58 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/13:0]
root 59 0.0 0.0 0 0 ? S Jan28 0:44 [ksoftirqd/13]
root 59006 0.0 0.0 0 0 ? S Sep06 0:00 [kworker/6:2]
root 60 0.0 0.0 0 0 ? S Jan28 0:48 [watchdog/13]
root 6 0.0 0.0 0 0 ? S Jan28 0:02 [migration/0]
root 61 0.0 0.0 0 0 ? S Jan28 0:58 [migration/14]
root 62 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/14:0]
root 62990 0.0 0.0 0 0 ? S Jul30 0:11 [flush-253:8]
root 63 0.0 0.0 0 0 ? S Jan28 0:56 [ksoftirqd/14]
root 64 0.0 0.0 0 0 ? S Jan28 0:47 [watchdog/14]
root 65 0.0 0.0 0 0 ? S Jan28 1:00 [migration/15]
root 654 0.0 0.0 0 0 ? S Jan28 0:03 [kjournald]
root 66 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/15:0]
root 67 0.0 0.0 0 0 ? S Jan28 0:56 [ksoftirqd/15]
root 68 0.0 0.0 0 0 ? S Jan28 0:46 [watchdog/15]
root 69 0.0 0.0 0 0 ? S< Jan28 0:00 [cpuset]
root 70 0.0 0.0 0 0 ? S< Jan28 0:00 [khelper]
root 7 0.0 0.0 0 0 ? S Jan28 0:46 [watchdog/0]
root 71 0.0 0.0 0 0 ? S< Jan28 0:00 [netns]
root 72 0.0 0.0 0 0 ? S Jan28 0:13 [sync_supers]
root 73 0.0 0.0 0 0 ? S Jan28 0:00 [bdi-default]
root 74 0.0 0.0 0 0 ? S< Jan28 0:00 [kintegrityd]
root 75 0.0 0.0 0 0 ? S< Jan28 0:00 [kblockd]
root 76 0.0 0.0 0 0 ? S< Jan28 0:00 [md]
root 77 0.0 0.0 0 0 ? S Jan28 0:00 [khungtaskd]
root 7723 0.0 0.0 0 0 ? S Sep04 0:04 [kworker/6:0]
root 79 0.0 0.0 0 0 ? S Jan28 0:00 [kswapd0]
root 80 0.0 0.0 0 0 ? SN Jan28 0:00 [ksmd]
root 8 0.0 0.0 0 0 ? S Jan28 0:03 [migration/1]
root 81 0.0 0.0 0 0 ? SN Jan28 0:28 [khugepaged]
root 82 0.0 0.0 0 0 ? S Jan28 0:00 [fsnotify_mark]
root 83 0.0 0.0 0 0 ? S< Jan28 0:00 [crypto]
root 87 0.0 0.0 0 0 ? S< Jan28 0:00 [kthrotld]
root 88 0.0 0.0 0 0 ? S< Jan28 0:00 [kpsmoused]
root 975 0.0 0.0 0 0 ? S Jan28 1:56 [kworker/1:2]
root 1858 0.0 0.0 4012 496 ? Ss Jan28 0:00 /sbin/acpid
root 8442 0.0 0.0 4504 676 tty1 Ss+ Jan28 0:00 /sbin/mingetty --noclear tty1
root 8443 0.0 0.0 4504 676 tty2 Ss+ Jan28 0:00 /sbin/mingetty tty2
root 8444 0.0 0.0 4504 676 tty3 Ss+ Jan28 0:00 /sbin/mingetty tty3
root 8445 0.0 0.0 4504 676 tty4 Ss+ Jan28 0:00 /sbin/mingetty tty4
root 8446 0.0 0.0 4504 676 tty5 Ss+ Jan28 0:00 /sbin/mingetty tty5
root 8447 0.0 0.0 4504 676 tty6 Ss+ Jan28 0:00 /sbin/mingetty tty6
XYZ 32034 0.0 0.0 4612 680 ? Ss 12:17 0:00 /usr/lib64/ssh/sftp-server
root 34828 0.0 0.0 4880 2468 ? S Apr29 0:33 /opt/uc4/uc4q/1/smgr/bin/../../agent/unix/bin/UCXJLI3
root 39577 0.0 0.0 4916 2376 ? S Jan28 3:50 /opt/uc4/uc4p/1/smgr/bin/../../agent/unix/bin/UCXJLI3
root 34826 0.0 0.0 4948 956 pts/0 R+ 12:27 0:00 ps aux
uuidd 7003 0.0 0.0 6072 0 ? Ss Jan28 0:00 /usr/sbin/uuidd -q -T 0
uc4sy 34825 0.0 0.0 6228 1248 ? S Apr29 0:40 /opt/uc4/uc4q/1/smgr/bin/ucybsmgr UC4Q1
uc4sy 7935 0.0 0.0 6228 816 ? S Jan28 1:07 /opt/uc4/uc4p/1/smgr/bin/ucybsmgr UC4P
root 4077 0.0 0.0 8372 4460 ? Ss Jan28 7:02 /sbin/haveged -w 1024 -v 1
root 7854 0.0 0.0 8560 644 ? Ss Jan28 0:00 /opt/perf/bin/ttd
root 4283 0.0 0.0 8772 536 ? Ss Jan28 32:58 /usr/sbin/irqbalance
root 2194 0.0 0.0 10160 4 ? Ss Jan28 0:00 /opt/hp/hp-health/bin/hp-asrd -p 1
root 2195 0.0 0.0 10292 468 ? S Jan28 3:01 /opt/hp/hp-health/bin/hp-asrd -p 1
root 1 0.0 0.0 10548 720 ? Ss Jan28 2:05 init [3]
XYZ 29516 0.0 0.0 13900 2664 pts/0 S 12:09 0:00 bash
root 29527 0.0 0.0 13908 2896 pts/0 S 12:09 0:00 bash
root 34827 0.0 0.0 13992 924 pts/0 S+ 12:27 0:00 sort -k 5n
root 8002 0.0 0.0 14100 3000 ? S Jan28 0:14 /opt/hps/inventory/python/bin/python2 /opt/hps/lw_agt/bin/lw_agt.py -f
root 4058 0.0 0.0 14436 592 ? S<sl Jan28 0:12 /sbin/audispd
root 7880 0.0 0.0 14620 2224 ? S<s Jan28 4:01 /opt/perf/bin/midaemon
root 7111 0.0 0.0 14800 556 ? Ss Jan28 0:20 /usr/sbin/cron
root 7124 0.0 0.0 16652 452 ? S Jan28 0:00 /usr/sbin/smartd
root 4173 0.0 0.0 16936 384 ? S< Jan28 0:00 /sbin/udevd --daemon
root 4174 0.0 0.0 16936 212 ? S< Jan28 0:00 /sbin/udevd --daemon
root 748 0.0 0.0 16940 460 ? S<s Jan28 0:00 /sbin/udevd --daemon
root 2061 0.0 0.0 18040 928 ? S Jan28 0:00 hald-runner
root 4133 0.0 0.0 18912 792 ? Ss Jan28 0:18 /sbin/rpcbind
root 5362 0.0 0.0 21292 640 ? S Jan28 1:00 cmaided -p 15 -s OK -l /var/log/hp-snmp-agents/cma.log
100 1871 0.0 0.0 21328 708 ? Ss Jan28 0:12 /bin/dbus-daemon --system
XYZ 29497 0.0 0.0 23160 2504 pts/0 Ss 12:09 0:00 -ksh
root 7205 0.0 0.0 23356 468 ? S Jan28 0:11 cmasm2d -p 30 -l /var/log/hp-snmp-agents/cma.log
root 5294 0.0 0.0 23360 508 ? S Jan28 0:35 cmasasd -p 15 -s OK -l /var/log/hp-snmp-agents/cma.log
root 5397 0.0 0.0 23364 584 ? S Jan28 0:35 cmascsid -p 15 -s OK -l /var/log/hp-snmp-agents/cma.log
root 5051 0.0 0.0 23624 964 ? S Jan28 19:04 cmahealthd -p 30 -s OK -t OK -i -l /var/log/hp-snmp-agents/cma.log
root 5052 0.0 0.0 23660 1080 ? S Jan28 11:38 cmastdeqd -p 30 -l /var/log/hp-snmp-agents/cma.log
root 6963 0.0 0.0 23804 356 ? S Jan28 0:00 /opt/hp/hpsmh/bin/rotatelogs /var/spool/opt/hp/hpsmh/logs/error_log 5M
root 6964 0.0 0.0 23804 356 ? S Jan28 0:00 /opt/hp/hpsmh/bin/rotatelogs /var/spool/opt/hp/hpsmh/logs/access_log 5M
root 5135 0.0 0.0 25588 1108 ? S Jan28 111:27 cmasm2d -p 30 -l /var/log/hp-snmp-agents/cma.log
root 5091 0.0 0.0 25980 776 ? S Jan28 4:45 cmathreshd -p 5 -s OK -l /var/log/hp-snmp-agents/cma.log
root 5164 0.0 0.0 26188 836 ? S Jan28 74:44 cmaperfd -p 30 -s OK -l /var/log/hp-snmp-agents/cma.log
root 4297 0.0 0.0 27112 372 ? Ss Jan28 0:00 /usr/sbin/mcelog --daemon --config-file /etc/mcelog/mcelog.conf
ntp 4980 0.0 0.0 28068 1276 ? Ss Jan28 9:42 /usr/sbin/ntpd -p /var/run/ntp/ntpd.pid -g -u ntp:ntp -i /var/lib/ntp -c /etc/ntp.conf
root 5067 0.0 0.0 28352 1344 ? S Jan28 33:52 cmahostd -p 15 -s OK -l /var/log/hp-snmp-agents/cma.log
root 4056 0.0 0.0 29648 340 ? S<sl Jan28 0:06 /sbin/auditd -s disable
101 2057 0.0 0.0 33920 2920 ? Ss Jan28 1:10 /usr/sbin/hald --daemon=yes
root 2156 0.0 0.0 34764 1648 ? S Jan28 0:00 hald-addon-input: Listening on /dev/input/event1
root 2175 0.0 0.0 34772 1868 ? S Jan28 0:00 /usr/lib64/hal/hald-addon-cpufreq
root 4119 0.0 0.0 35596 1656 ? Sl Jan28 0:00 /opt/opsware/agent/bin/python /opt/opsware/agent/pylibs/shadowbot/daemonbot.pyc --conf /etc/opt/opsware/agent/agent.args
101 2177 0.0 0.0 36520 1544 ? S Jan28 0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
root 5200 0.0 0.0 38332 972 ? Sl Jan28 0:54 cmapeerd -l /var/log/hp-snmp-agents/cma.log
root 7873 0.0 0.0 39296 5916 ? Ss Jan28 3:53 /opt/perf/bin/perfalarm
root 7084 0.0 0.0 40244 1996 ? Ss Jan28 0:31 /usr/lib/postfix/master
postfix 24260 0.0 0.0 42100 2288 ? S 11:50 0:00 pickup -l -t fifo -u
postfix 7110 0.0 0.0 42260 2104 ? S Jan28 0:05 qmgr -l -t fifo -u
root 7866 0.9 0.0 43036 11776 ? S<s Jan28 3043:56 /opt/perf/bin/scopeux
root 29525 0.0 0.0 43132 3312 pts/0 S 12:09 0:00 sudo /webdienste/../bin/su
root 5225 0.0 0.0 51104 2252 ? Sl Jan28 2:07 cmaeventd -p 15 -l /var/log/hp-snmp-agents/cma.log
root 4959 0.0 0.0 51420 928 ? Ss Jan28 0:00 /usr/sbin/sshd -o PidFile=/var/run/sshd.init.pid
root 5328 0.0 0.0 51524 2484 ? Sl Jan28 21:24 cmaidad -p 15 -s OK -l /var/log/hp-snmp-agents/cma.log
root 5259 0.0 0.0 53276 2376 ? Sl Jan28 98:42 cmafcad -p 15 -s OK -l /var/log/hp-snmp-agents/cma.log
root 7360 0.0 0.0 54532 7152 ? Sl Jan28 0:00 /opt/OV/lbin/eaagt/opcmsgi
root 7313 0.0 0.0 68144 7576 ? Sl Jan28 17:31 /opt/OV/lbin/eaagt/opcmona
root 41603 0.0 0.0 70572 6828 ? SNl Sep07 0:20 /usr/sbin/snmpd -r -A -LF i /var/log/net-snmpd.log -p /var/run/snmpd.pid
root 29526 0.0 0.0 75380 2580 pts/0 S 12:09 0:00 /webdienste/../bin/su
hpsmh 6968 0.0 0.0 85924 660 ? S Jan28 0:27 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf/smhpd.conf
hpsmh 6969 0.0 0.0 85924 640 ? S Jan28 0:00 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf/smhpd.conf
hpsmh 6970 0.0 0.0 85924 640 ? S Jan28 0:00 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf/smhpd.conf
hpsmh 6971 0.0 0.0 85924 640 ? S Jan28 0:00 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf/smhpd.conf
hpsmh 6972 0.0 0.0 85924 640 ? S Jan28 0:00 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf/smhpd.conf
root 6962 0.0 0.0 85924 1220 ? Ss Jan28 4:41 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf/smhpd.conf
root 5418 0.0 0.0 87688 2756 ? Sl Jan28 0:50 cmanicd
root 31939 0.0 0.0 92452 3708 ? Ss 12:17 0:00 sshd: XYZ [priv]
root 29403 0.0 0.0 92664 3736 ? Ss 12:09 0:00 sshd: XYZ [priv]
XYZ 29496 0.0 0.0 92664 2168 ? S 12:09 0:00 sshd: XYZ@pts/0
XYZ 32033 0.0 0.0 92808 2352 ? S 12:17 0:00 sshd: XYZ@notty
root 7317 0.0 0.2 96356 37492 ? Sl Jan28 23:34 /opt/OV/lbin/eaagt/opcle -std
root 2060 0.0 0.0 98600 1104 ? Ssl Jan28 0:03 /usr/sbin/console-kit-daemon
root 4091 0.0 0.0 100640 4300 ? SLl Jan28 10:01 /sbin/multipathd
root 7283 0.0 0.0 120892 10020 ? Sl Jan28 0:08 /opt/OV/lbin/conf/ovconfd
root 7342 0.0 0.0 121504 8060 ? Sl Jan28 19:13 /opt/OV/lbin/eaagt/opcacta
root 41261 0.0 0.0 122032 1280 ? Sl Jul15 0:05 /sbin/rsyslogd -c 5 -f /etc/rsyslog.conf
root 7256 0.0 0.0 154744 9240 ? Ssl Jan28 51:58 /opt/OV/bin/ovcd
root 1970 0.0 0.0 160808 864 ? Ssl Jan28 39:03 hpasmlited -f /dev/hpilo
root 7363 0.0 0.1 161408 20296 ? Sl Jan28 4:32 /opt/OV/lbin/perf/coda
root 7264 0.0 0.0 170332 8612 ? Sl Jan28 1:48 /opt/OV/bin/ovbbccb -nodaemon
root 6982 0.0 0.0 199220 1184 ? Ssl Jan28 5:29 /usr/sbin/nscd
root 25142 0.0 0.0 208656 10604 ? Sl Jun07 4:33 /opt/OV/lbin/eaagt/opcmsga
root 4130 0.0 0.2 313492 35972 ? Sl Jan28 40:40 /opt/opsware/agent/bin/python /opt/opsware/agent/pylibs/shadowbot/daemonbot.pyc --conf /etc/opt/opsware/agent/agent.args
root 24359 0.2 27.7 5528980 4413908 ? Sl Aug19 61:25 /webdienste/java/jdk/ABCD2.com/current/bin/java -Djava.util.logging.config.file=/webdienste/appserv-jsp/tomcat/ABCD2.com/current/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Xms4096m -Xmx4096m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:MaxGCPauseMillis=1500 -XX:GCTimeRatio=9 -server -XX:+DisableExplicitGC -Djava.endorsed.dirs=/webdienste/appserv-jsp/tomcat/ABCD2.com/current/endorsed -classpath /webdienste/appserv-jsp/tomcat/ABCD2.com/current/bin/bootstrap.jar:/webdienste/appserv-jsp/tomcat/ABCD2.com/current/bin/tomcat-juli.jar -Dcatalina.base=/webdienste/appserv-jsp/tomcat/ABCD2.com/current -Dcatalina.home=/webdienste/appserv-jsp/tomcat/ABCD2.com/current -Djava.io.tmpdir=/webdienste/appserv-jsp/tomcat/ABCD2.com/current/temp org.apache.catalina.startup.Bootstrap start
root 47962 0.2 27.7 5574472 4410676 ? Sl Aug12 77:49 /webdienste/java/jdk/ABCD2.com/current/bin/java -Djava.util.logging.config.file=/webdienste/appserv-jsp/tomcat/ABCD2.com/current/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Xms4096m -Xmx4096m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:MaxGCPauseMillis=1500 -XX:GCTimeRatio=9 -server -XX:+DisableExplicitGC -Djava.endorsed.dirs=/webdienste/appserv-jsp/tomcat/ABCD2.com/current/endorsed -classpath /webdienste/appserv-jsp/tomcat/ABCD2.com/current/bin/bootstrap.jar:/webdienste/appserv-jsp/tomcat/ABCD2.com/current/bin/tomcat-juli.jar -Dcatalina.base=/webdienste/appserv-jsp/tomcat/ABCD2.com/current -Dcatalina.home=/webdienste/appserv-jsp/tomcat/ABCD2.com/current -Djava.io.tmpdir=/webdienste/appserv-jsp/tomcat/ABCD2.com/current/temp org.apache.catalina.startup.Bootstrap start
ABCD2:/webdienste/appserv-jsp/tomcat/ABCD2.com/current #

****

Tim Holloway wrote:At the OS level, you might want to set up an ongoing vmstat whose results are collected to a file and then plotted with gnuplot or something like that (assuming you'd like graphs instead of just raw numbers). VMStat can help detect "spikes" in resource demands over time, where "top" displays only the demand of the instant.

This looks like it might help you understand how vmstat is used: http://www.admin-magazine.com/HPC/Articles/Stat-like-Tools-for-Admins

At the JVM level, try running the jprofile tool to see how it's using memory internally. Considering all the grief, it might be worth spending some money on some commercial analysis tools.

Hi All,

Since yesterday, the application performance on Tomcat AS 7 is again very sluggish:

ABCD@serverABCD:~> uname -a
Linux serverABCD 3.0.101-0.35-default #1 SMP Wed Jul 9 11:43:04 UTC 2014 (c36987d) x86_64 x86_64 x86_64 GNU/Linux
ABCD@serverABCD:~> date
Tue Sep 8 09:45:10 CEST 2015
ABCD@serverABCD:~> ps -ef |grep java |grep ABCD2
root 24359 1 0 Aug19 ? 01:01:04 /webdienste/java/jdk/ABCD2.intranet.ABCD.com/current/bin/java -Djava.util.logging.config.file=/webdienste/appserv-jsp/tomcat/ABCD2.intranet.ABCD.com/current/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Xms4096m -Xmx4096m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:MaxGCPauseMillis=1500 -XX:GCTimeRatio=9 -server -XX:+DisableExplicitGC -Djava.endorsed.dirs=/webdienste/appserv-jsp/tomcat/ABCD2.intranet.ABCD.com/current/endorsed -classpath /webdienste/appserv-jsp/tomcat/ABCD2.intranet.ABCD.com/current/bin/bootstrap.jar:/webdienste/appserv-jsp/tomcat/ABCD2.intranet.ABCD.com/current/bin/tomcat-juli.jar -Dcatalina.base=/webdienste/appserv-jsp/tomcat/ABCD2.intranet.ABCD.com/current -Dcatalina.home=/webdienste/appserv-jsp/tomcat/ABCD2.intranet.ABCD.com/current -Djava.io.tmpdir=/webdienste/appserv-jsp/tomcat/ABCD2.intranet.ABCD.com/current/temp org.apache.catalina.startup.Bootstrap start
ABCD@serverABCD:~>

ABCD2 instance runs since 19th August 2015 (and no reboot was performed recently).

Both the OS platform as well as the Web Components have been normal – yesterday and today – no suspicious load on server:

09:51:55 up 222 days, 21:47, 1 user, load average: 0.27, 0.21, 0.16
USER TTY LOGIN@ IDLE JCPU PCPU WHAT
ABCD pts/0 09:36 3.00s 0.01s 0.00s w

Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 15529M total, 15302M used, 226M free, 365M buffers
Swap: 16383M total, 86M used, 16297M free, 5868M cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24359 root 20 0 5399m 4.2g 10m S 1 27.8 61:05.45 java

---

Only point of concern is the used memory:

Mem: 15529M total, 15302M used, 226M free, 365M buffers

Any pointers?

Tim Holloway wrote:The PermGen size isn't likely to affect performance. You generally know when it's too small when you get an out of PermGen Space exception, which is most likely to happen when you've got a complex application with stuff like ORM and lots of servlets. It also tends to show up when you repeatedly re-deploy a webapp without restarting Tomcat, since the old version's PermGen space isn't completely recovered.

Java 8 gets rid of the whole PermGen thing, incidentally.

so you are suggesting, if the DNS name of the SFTP (MFT) server changes, we do not need to change anything at the client side other than hostname change?

But then what are these client certificates for used by the client tool while establishing the connection?

Eignes/Own Certificate (.p12) and MFT certificate (.p7b)

To address the requirements a product named MeRLin from a provider named abc is used. MeRLin is a client application that is installed on end-user machines with a file-based data store sitting in a shared drive. The client exchanges messages with the external party using Managed File Transfer (aka MFT, Secure Transport). The MFT is hosted in a DMZ and is exposed over internet to a server sitting on external party side. Client communicates with MFT using SSH over FTP protocol and authenticates with it using certificates. The MFT in turn transmits the encrypted messages to external party server over a secured channel using certificates. Once messages are read by client, they are removed from MFT.

A.J. Côté wrote:

Norman Meister wrote:Thanks for replying.


The client tool needs to be configured with:

Hostadresse: mft.intranet.ABC.com Port: 22
Username: YYY
Password: ***
Remote-Path: /toAmp or /fromAmp
Host-Key: “sshd.pub“ import

The Merlin Client also uses an Eignes/Own Certificate (.p12) and MFT certificate (.p7b) in order to complete the connection setup with MFT server.

In the future, Hostadresse would be lets say: mft.intranet.com Port: 22 i.e. without ABC.

Based on the information you provide, it should work without any changes on the server nor client except the server host address on the client of course.

Thanks for replying.


The client tool needs to be configured with:

Hostadresse: mft.intranet.ABC.com Port: 22
Username: YYY
Password: ***
Remote-Path: /toAmp or /fromAmp
Host-Key: “sshd.pub“ import

The Merlin Client also uses an Eignes/Own Certificate (.p12) and MFT certificate (.p7b) in order to complete the connection setup with MFT server.

In the future, Hostadresse would be lets say: mft.intranet.com Port: 22 i.e. without ABC.

A.J. Côté wrote:SFTP usually uses public/private key pair, no certificate per say. Are you sure your client is using a certificate? Even then, the public/private key pair of the client should be retrieved from the certificate and it shouldn't make any difference.

It should be transparent.

If you changed the public/private key pair on the server without changing the hostname then you should get a warning since clients usually save the key fingerprint of the server to prevent man in the middle attacks but you would still be able to connect.

Hi All,

I have an SFTP (FTP over SSH) client that communicates with an SFTP server (MFT, Secure Transport from Axway). The SFTP (MFT) server is also hosted within my company (intranet).
The client application uses a certificate to establish a connection with the SFTP (MFT) server.

The SFTP (MFT) server has a domain name: mft.intranet.ABC.com Port: 22.

In the future the SFTP host name is going to be changed to: mft.intranet.XYZ.com Port: 22.

I have 2 questions:

Does the SFTP server uses a certificate? If yes, after the change in its DNS name, do we need to get a new certificate for the server?
Does the client also need to change the client certificate, since it now communicate with the same server but with different DNS?

Thanks.

Thanks, will install the JProfile on the server.

free -m
total used free shared buffers cached
Mem: 7986 6212 1774 0 767 2857


I am also planning on setting the setenv to:

export CATALINA_OPTS="$CATALINA_OPTS -Xms4096m"
export CATALINA_OPTS="$CATALINA_OPTS -Xmx4096m"

# Increase maximum perm size for web base applications to 4x the default amount
# http://wiki.apache.org/tomcat/FAQ/Memoryhttp://wiki.apache.org/tomcat/FAQ/Memory
export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxPermSize=512m"

export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelGC"
export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=1500"
export CATALINA_OPTS="$CATALINA_OPTS -XX:GCTimeRatio=9"
export CATALINA_OPTS="$CATALINA_OPTS -server"

# Disable remote (distributed) garbage collection by Java clients
# and remove ability for applications to call explicit GC collection
export CATALINA_OPTS="$CATALINA_OPTS -XX:+DisableExplicitGC"

Tim Holloway wrote:Hmmm. About 4GB of totally unused RAM, it appears.

At this point, I'd be inclined to suspect that either your webapps are running short but intense bouts of resource usage - either something really compute-heavy or relying on some external resource that has a long latency.

Probably the best approach would be to run a profiler on the webapp and see where most of its time is being spent.